BEST AVAILABLE SCIENCE:

METRICS FOR EVALUATION OF SCIENTIFIC CLAIMS

For several decades, the staff of the Institute for Regulatory Science has been involved in the development of a process to evaluate the validity of scientific claims. We chose the term Best Available Science (BAS) to generically sort out the validity of claims and counter claims. These efforts led to the development of Metrics for Evaluation of Scientific Claims (MESC). The BAS concept uses five Principles for evaluation a scientific claim leading to three pillars.

BAS PRINCIPLES

Those who attempt to search the relevant literature would benefit from the availability of principles that provide fundamental guidance for the search process.

Open-mindedness and Skepticism: The Open-mindedness Principle complements the Skepticism Principle. Whereas the Open-mindedness Principle implies the willingness to accept new knowledge, the Skepticism Principle requires that those who make a scientific claim must provide sufficient evidence supporting their claim.

Universal Rules: This principle--previously referred to as Universal Scientific Principles--implies that all scientific disciplines use certain methods, processes, and techniques in pursuit of their professional activities. For example, all scientific disciplines use specific computational methods and apply the rules of statistics in sampling, analysis, and reporting their results. Let us use the example of a reaction. If someone claims a reaction, it would require the demonstration that an action caused the reaction.

Transparency: One of the primary reasons for scientific controversies is the assumption of policy makers-- including regulators--that the public is incapable of comprehending the complexities of various branches of science. If a scientific issue is complex, it is the responsibility of regulatory scientists to explain the subject to the public in a language that is understandable to its recipients.

Reproducibility Principle: The ultimate proof of the validity of a scientific claim that it is reproducible by those who have the necessary competency and the needed equipment and facilities. This principle separates undisputed areas of science from those that include assumptions, interpretations, and in some cases, the inclusion of ideological and societal objectives in a scientific assertion.

 

PILLAR: CLASSIFICATION OF SCIENCE

This pillar identifies the level of maturity of Science published in scientific journals; reports prepared by government agencies; pronouncements of professional organizations; opinions expressed by advocacy groups; or any other scientific expression.

 

Proven Science

This group consists of scientific laws (sometimes called scientific principles); application of scientific laws; and virtually proven science.

Scientific Laws: Scientific laws are unequivocally confirmed and generally accepted. The cornerstone of this class is that any investigator who has the necessary skills and proper equipment can reproduce it. Similarly, this class of science does not require assumptions or any other prerequisite for its proof.

Applied Science: This class consists of application of scientific laws to various branches of commerce and industry. The only prerequisite for a scientific claim to be placed into this class is the correct interpretation of scientific laws. Much of the Engineering disciplines and a large segment of other applied sciences fall into this class.

Virtually Proven Science: This class consists of scientific information whose reliability has been uncontested yet there is insufficient proof to be placed in the proven science class. The process for placing science into this class consists of: 1) ensuring that no reliable observation contradicts information in this class and 2) there is no alternative system that describes the observed phenomena. Possible examples of this class include both general and special relativity theories. These theories attempt to describe the universe, forces operating in the universe, and their interrelationship.

 

Evolving Science

The overwhelming majority of scientific advances in virtually all disciplines are evolving science.

Reproducible Evolving Science: Reliable scientific assertions dealing with a subject that is not completely understood constitutes the core of this class. The key factor in placing a scientific assertion into this category is reproducibility. Although the observations that are the foundation of this class may not necessarily have general applicability, they are clearly and unambiguously reproducible by those with appropriate skills and equipment. Advancements in virtually all branches of science including physics, chemistry, biology, and many other scientific disciplines are based on the desire of investigators to improve knowledge in this class.

Partially Reproducible Evolving Science: The scientific foundation of scientific claims in this class is derived from proven or reproducible evolving science. However, it uses assumptions, extrapolations, and other similar processes in deriving its results and conclusions. A characteristic of this class is that the investigator who is trying to reproduce this class must not only have proper skills and the necessary equipment, but must also accept the asserted scientific foundation, assumptions, choice of mathematical processes, default data, and numerous other prerequisites. Most predictive models and a large segment of contested scientific information fall into this class.

Association-Based Evolving Science: Sometimes called correlation or observation studies, this class is not based on Proven or Reproducible Evolving Science. Instead, often an investigator attempts to correlate two groups such as two populations with different characteristics hoping to correlate an observed difference between the two populations to a cause. Consequently, the level of reproducibility of information in this class ranges from unknown to reasonable.

Hypothesized Science: This class consists of an organized response to an observation, an idea, or any other initiating thought process. Experience shows that although many great scientific discoveries started with this class, there is also a long list of hypothesized claims that have proven to be either wrong or not worth pursuing.

 

Borderline Science

As the title implies, Borderline Science is exactly that and should be used accordingly.

Judgment: On occasion, information is provided to the society that lacks scientific foundation. Similarly, often decisions must be made without having the needed scientific information including basic principles, the necessary data, and other scientific requirements. The methodology for expert judgment is reasonably well-developed and consists of asking a number of presumably knowledgeable individuals to give answers to specific questions and statistically assess the results. However, this class is often an educated guess.

Speculation: This class consists of information that cannot meet standards described in any of the above classes. It is often based on the intuition of an individual who wants to stimulate a discussion or initiate a research project.

 

Fallacious Information

Information that clearly falls into the purview of science but is inconsistent with the three groups of classes identified above is fallacious information. This information is often called “pseudo science”, “junk science”, or “politically-processed science”.

 

PILLAR: CATEGORIZATION OF RELIABILITY OF SCIENCE

There are rational and reasonable uncontested methods to resolve scientific controversies. Briefly, scientific claims or assertions are divided into four distinct categories. For the sake of simplicity, in the following the description of each group starts from the bottom of the figure.

Personal Opinions: Expression of views by individuals regardless of their training, experience, and social agenda are included in this group. Personal opinions are seldom, if ever, BAS. At best, this category can be used to initiate the study of a scientific issue. Note that the standard process of the news media is reliance upon this category in its reporting of scientific issues.

Gray Literature: This is the favorite category of government agencies, advocacy groups, and individuals who want to promote an idea. In fact, this category is the more organized and written form of personal opinions. Again here, at best, this category should be used to initiate a study. Experience shows that in the overwhelming majority of cases this category does not meet the requirements of scientific acceptability.

Peer Reviewed Science: A scientific claim subjected to an independent peer review constitutes this category. Peer review is the foundation of scientific acceptability. There are numerous requirements for acceptability of peer review. Briefly, the individual who is chosen as a reviewer must be a “peer” to the author of the study, and must have no conflict of interest. In addition, the author of the study must respond to criticism by the peer to the satisfaction of an uninvolved person or organization.

Consensus-Processed Science: This category uses the consensus process providing an opportunity to resolve scientific disputes. The prerequisite for this process is identical to the process used for independent peer review.

 

PILLAR: AREAS OUTSIDE THE PURVIEW OF SCIENCE

The concept of BAS deals with classification of scientific information and evaluation of scientific assertions. It specifically excludes areas that are outside of science. The inclusion of faith, ideology, societal objectives, beliefs, or any other non-scientific objectives in assessing the validity of scientific information is inconsistent with the foundation of BAS.