Philosophical Multicore

Sometimes controversial, sometimes fallacious, sometimes thought-provoking, and always fun.

The Credibility Spectrum

Posted by Michael Dickens on July 19, 2009

Inspired by this and this, I am going to design my own climatological credibility spectrum. When it comes to global warming, how much weight should be given to each person’s word? I am going to name the different types of people and organizations, as well as how much weight I think they should be given. The weightings are mere estimates, so don’t take them too seriously.

Individuals

layperson = 1
professional = 3
scientist = 10
earth scientist = 20
climatologist = 30

Is the scientist actively publishing? If so, multiply by three.

Organizations

Petition = 50 * ln((number of people in the petition) – 200)
Justification: Petitions with less than 200 people are not considered. It is logarithmic because as you continue to add more people, the petition only becomes so much more reliable. For example, if the petition has 300 people, the value of the petition is 230. If the petition has 600, the value is 285. 1000 people equate to a value of 328. A million people are valued at 691.

University = 3000, more or less. It depends on the size of the university.

Peer reviewed paper = 5000

Scientific organization = 100 * square root (number of members)
Justification: The weighting does not increase substantially as the organization gets larger, but it is still significantly more credible than a petition. AAAS, with about 130,000 members, has a weighting of about 36,000. An organization with 20,000 members is worth 14,000; 5000 equates to 7000.

Actually, the weighting for a scientific organization doesn’t make much sense; for larger organizations, a single climatologist is worth about a one hundredth as much as a climatologist outside of a scientific organization. But after all, the opinion of a scientific organization doesn’t necessarily represent the opinion of every single person inside of the organization, so I suppose it’s fair.

Thoughts? Feedback? How can I improve the weightings?

Advertisements

12 Responses to “The Credibility Spectrum”

  1. BobC said

    WTF?
    Peer reviewed paper = 5000 !?

    Have you ever participated in the peer review process?

    If it’s this good, we should use it for everything. For instance, if a company wants to dig an open pit mine, we should only let other open pit mine owners make the decision on whether to issue a permit. It’s peer review — how could that be misused?

    Where do verifiable data fit into your scheme? That’s the bottom line, after all — you know, reality.

    • mtgandp said

      Well as I said, these are only rough estimates. And this isn’t designed to be a credibility spectrum for just anything; it’s more for scientific topics. Peer review does not work for everything.

      I’m not sure what you mean by verifiable data.

    • mtgandp said

      Well then, it’s about time that we got an independent audit on climate change!

  2. BobC said

    I’m not sure what you mean by verifiable data.

    That’s a fair question. A comprehensive answer is difficult (or impossible), but one can make judgements in specific situations. In general, data has more credibility if it has been produced by:

    a) More than one technical method,

    and/or

    b) More than one group of persons, with diverse biases and funding sources,

    and/or

    c) Work that has been independently audited by a group (perhaps government) that does not share the same goals and motivations as the group producing the data. (This is how important software is verified, and how the FAA, FDA, and other regulatory agencies are supposed to work. Most people would not be satisfied if the FDA, say, based their decisions only on the opinions of the drug company’s scientists or their publications.)

    Data that has survived the above sorts of tests has more credibility than data from a single source, or data that has been produced using proprietary means.

    The problem with peer review is that it represents a very narrow (and small) consensus of individuals with similar biases and outlooks. For example, for a paper to be published in a journal, it must pass the following tests:
    1) The editor of the journal wants to publish it for reasons often not made public. It might be that the paper represents an important advance in the editor’s opinion — or it might fit in with one of the editor’s biases.
    2) Two or three individuals (the reviewers), picked by the editor, agree.

    The identities of the reviewers are never made public, so there is no way to know if there are conflicts of interest involved (sharing, or competing for funding sources, for example, or a history of reviewing each other’s papers). The reviewers are never given enough data or time to actually determine if the author is correct (if that is even possible), so the best that can be done is to see if there any obvious logical errors or insufficient reference to prior work. At it’s worst, peer review can operate like an “Old boy’s network” where the authors and reviewers collectively promote their own work and theories and exclude competitors. (Sort of how most everyone would expect the mine owners to behave — why we don’t allow peer review for those kinds of decisions.)

    Peer review absolutely does not provide any guarantee of correctness (as is proven by the ease with which peer reviewed papers can be found that contradict each other).

    • mtgandp said

      You make an interesting point. I think that mentioning verifiable data at all begs the question, since this credibility spectrum is a way of finding verifiable data.

      While peer review may not be perfect, it is still more reliable than the opinion of some person or some group of people. Perhaps 5000 is too high a weighting, but it should definitely be higher than the opinion of a climatologist or a small group of climatologists.

      • BobC said

        “…it is still more reliable than the opinion of some person or some group of people.”

        Why would the opinions of an anomymous group of people (the reviewers) be more reliable than the opinion of known persons whose backgrounds and qualifications are available?

        In fact, the reliability of an anomymous group is unknowable.

      • mtgandp said

        Peer review is especially designed to remove bias. Normal people in a group are not checking for biases in each other’s opinions.

  3. BobC said

    “Peer review is especially designed to remove bias.”

    I must admit, I’m at a complete loss to understand your reasoning — perhaps you could explain it?

    In my experience (and I don’t think it’s unique), people are much more likely to express and act on bias anomymously than they are when they can be held to account. This is why we seek transparency in government actions and put limits on the actions of lobbyists.

    If peer review were designed to remove bias, the signed reviews would be published with the paper so that potential conflicts of interest and biased reviews were plainly evident and could be critized.
    (This would also make it much harder for editors to find sufficient willing reviewers.)

    I was once, by chance, asked to review a paper by a scientist who had previously launched a personal and vehement attack on me at a meeting where I was giving a talk. He went even further, and tried to have my company’s funding (a contract for an instrument) shut down. I was able, subsequently, to demonstrate publicly that his objection was based on an elementary mistake (much to his chagrin). The rules of the journal (Optical Engineering) precluded me from signing the review, so I made specific reference to the incident so that he would be able to identify me. I gave him a fair review (recommending publication), but I also, I hope, gave him a bit of a warning to play nicer in the future.

    What if he had succeeded in stopping my funding? Would I have succumbed to the temptation to black-ball him? Maybe — I’m not a saint.

    Obviously, the most ethical action for me to have taken would have been to decline to review the paper — which, in fact, I would have done had the review been public. If my honest opinion had been to not publish, I think I also would have declined, just to avoid the appearance of bias. Since I thought the paper was good, however, I couldn’t resist privately tweaking his tail a little.

    Reviewing papers is a pain — it takes a lot of effort and time. If you show any willingness to do it at all, you are innundated with requests from editors. Since editors try to publish papers at the leading edge of their field, the number of potential reviewers is necessarily limited to others at or near that particular edge of technology. In this situation, it is very easy for a group of people to end up reviewing each other’s papers. If those people share the same funding source (not that uncommon with government funding) then you have an essentially closed clique, with the same goals and biases, self-validating each other.

    This is precisely why the FDA (or the FAA, etc) goes to outside sources when seeking to validate claims, rather than just polling the company’s scientists.

    In fact the principle of peer review is, in nearly any other realm of Human behavior, immediately recognized as highly vulnerable to corruption.

    As an engineer, I have often used results from the peer reviewed literature in projects. It is startling how often (~20% of the time) you find that the results are flawed or wrong and won’t work in the real world.

    In science and technology, independent replication is the best guarentee of correctness, folowed by independent audits and testing (with stress on “independent”). Peer review comes in way down the list.

    • mtgandp said

      You make a good point.

      “I must admit, I’m at a complete loss to understand your reasoning — perhaps you could explain it?”
      If not to check for errors and eliminate bias, what is the purpose of peer review? Even if the purpose is not upheld, it still must help to at least be TRYING to reduce errors/bias, even if it doesn’t always work out in practice.

      So what weighting would you give to peer reviewed papers?

      • BobC said

        Mtgandp;

        Wikipedia has a pretty good article on peer review ( http://en.wikipedia.org/wiki/Peer_review ), including many of its advantages and weaknesses.

        You are right in that the purpose of peer review is to reduce the possibility of error, especially errors that might be embarrassing to the journal or damage its reputation.

        I have no objection to this use of peer review. That is because, until recently, peer reviewed publications weren’t very important. They are important to the author’s reputation, of course, and can affect the journal’s reputation and hence its profitability. But that was about it.

        Discoveries and scientific work with large economic potential are always protected either through the patent system, or are kept proprietary. (The main motivation for the patent system is to reward inventors — with a temporarily monopoly — for revealing their inventions, thus increasing the eventual benefit to Humanity. It has the additional advantage for scientists that, after the IP department is satisfied with the patent application, they can go ahead and publish the results in a journal.)

        (Research without immediate economic potential can still be important, of course. But in that case there is usually plenty of time for others to confirm or falsify the research through replication.)

        Government regulation often uses an audit system to establish probable truth. In the Wiki article:
        “A more rigorous standard of accountability is known as an audit. Because reviewers are not paid, they cannot be expected to put as much time and effort into a review as an audit requires. Most journals (and grant agencies like NSF) have a policy that authors must archive their data and methods in the event another researcher wishes to replicate or audit the research after publication[citation needed]. Unfortunately, the archiving policies are sometimes ignored by researchers.”

        Because of the uneven quality of peer review, it is not wise for the government to make serious policy decisions on the basis of peer-reviewed research. Nearly all regulatory agencies rely instead on audited or replicated research.

        Recently the EPA attempted to suppress an internal report calling for an audit of climate science before making any regulations ( http://cei.org/news-release/2009/06/25/cei-releases-global-warming-study-censored-epa ). If the issue was really the environment, this should have been a standard procedure as it has many times in the past.

        The entire AGW debate could be put to rest with an independent audit, whichever way it went — and the EPA is an ideal organization to conduct it. Since the only real problem with an audit (for either side) is that it might not support your position, I find it significant that, by and large, the anti-AGW side (the “deniers”) are OK with an audit, and the pro-AGW side is not. My tentative conclusion from this is that the pro-AGW side’s motives are primarily political, not concern for the environment.

        In answer to your question: Unless serious bias is uncovered in the journals, I would accept that peer reviewed papers be primarily considered for an audit. I would give unaudited or unreplicated papers a very low weight, regardless of peer review status.

  4. Roy Latham said

    Scientific truth is not determined by polls or surveys, no matter how they are weighted. Your whole approach is wrong, because it is just an embellishment of the notion that the popularity of belief reflects its validity. A few decades ago, there was a broad consensus supported by peer-reviewed literature and the official statements of professional societies that homosexuality was a form of mental illness. Somehow, that didn’t end the issue. There are innumerable parallels, like Newtonian Mechanics and the Steady State Theory of the universe. Einstein’s original publications were not peer reviewed.

    The test of scientific theory is its ability to predict future events. CO2 crisis theory says that CO2 has for the past few decades been dominating climate. Temperature increased with CO2 from the late 70s to the mid-90s. consistent with the theory. Since the mid-90s, CO2 has continued to increase, but temperature has stalled. It remains warm, but there is no trend. Therefore the theory that nothing is at work but CO2 is dead.

    As to peer review, read “The Hockey Stick Illusion” as to how a completely invalid paper passed peer review. The basic error was in the use of statistics, and the Hockey Stick paper was kept out of the hands of experts in statistics. Only those who agreed with the conclusion were included in the review. Climategate showed that CO2-crisis proponents are eager to control journals in a way to ensure that skeptics don’t get published.

    • As to the determination of scientific truth, you are of course completely correct. The relevant point here isn’t that scientists are necessarily right, but that they’re more likely to be right than anyone else. In day to day life, no one has time to investigate every claim they come across. So it’s useful to have an easy and fairly reliable method of deciding whether to believe a claim: if it came from a scientist, it’s probably true. That’s not to say that it is true, but it’s more likely to be true than if some random person told you.

      I don’t want to get in an argument about global warming, but you made one point that I feel I have to address. You claim that temperature has not increased since the mid-90s (I’ve heard you give 1998 as the exact date). But according to NASA GISS, temperature has clearly been increasing over the last twelve years, and 2010 is well on the way to being the warmest year on record.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: