Purpose: SETAC is a professional society with worldwide membership from academia, government, business, and nongovernmental organizations. TIPs provide a credible and balanced scientific discussion of important environmental issues.
Sound Science
“Sound science” is a popular term that means different things to different people. When used to support a position on a controversial issue or decision, sound science generally means “I’ve got the facts on my side.” But often questions are raised such as: Whose facts? What is their source? Are the data reliable? Is the interpretation biased?
The fast pace of today’s science and its role in making important decisions makes it imperative that policy-makers, the media, and the general public are able to distinguish the facts from mere interpretations of a biased constituency. Decision-makers and those who inform them must be able to judge the quality of the science and reasoning that supports a position and must know whether a set of scientific findings is really meaningful to a decision.
This Technical Issue Paper (TIP) was developed by a group of scientists from academia, government, and business, under the auspices and support of the Society of Environmental Toxicology and Chemistry. The TIP describes what sound science means and does not mean, describes ways in which sound science is used to set public policy, and offers some practical suggestions for judging the adequacy of technical studies and their interpretation.
What is sound science?
Sound science can be described as organized investigations and observations conducted by qualified personnel using documented methods and leading to verifiable results and conclusions.
Sound science implies that a set of data, facts, or conclusions of a scientific nature are supported by studies that follow the high standards of the scientific method. These standards describe important investigational attributes and practices such as the formulation of a readily testable hypothesis; the use of systematic and well-documented experimental or analytical methods (e.g., adequate sample sizes, appropriate control experiments); the application of appropriate data analysis tools (e.g., statistics and mathematical models) to the data; and the articulation of conclusions that address the hypothesis and are supported by the results. The scientific method is broadly applicable to various types of investigations including basic research, applied research, theoretical conjectures, descriptive studies of nature, and technology applications such as environmental monitoring and literature reviews. The scientific method helps to ensure that the investigations and observations are sound, i.e., that the data and results are reliable and the conclusions are supported by the data.
Sound science should not be confused with facts or information that are generally accepted or readily believable. Major breakthrough discoveries in science over the centuries have at times conflicted with prevailing wisdom and beliefs. The biochemist Hans Krebs (who described a major biochemical process that cells use to convert energy) and the geneticist Barbara McClintock (who discovered that genes may change locations on a chromosome) are among the Nobel Laureates whose work was initially dismissed by colleagues as too radical. As their work was carefully reviewed, reproduced by other scientists, and verified by other approaches and independent lines of evidence, their findings were indeed found to stand the test of time.
At the same time, sound science should not be taken to imply that the findings will stand forever as the absolute truth. The development of improved analytical tools and the radical expansion of technology often require that old models and beliefs be revised or even discarded. This is as true today as it was when Copernicus startled the medieval world by concluding that the earth was not the center of the universe. New observations and discoveries in astronomy and physics continue to alter the way scientists view the origin and structure of the universe. The same is true in other sciences as well.
Given the likelihood that new understanding will replace old beliefs, will we ever truly know whether a study represents sound science? Even in the absence of consensus, individual studies still may be evaluated on their individual merits by questions such as: Are they consistent with the scientific method? Did they use documented methods? Were the investigators qualified? Too often, advocates of a particular issue leap upon news media reports of scientific studies that seem to support their argument. Until others in the field have an opportunity to review the work, it is premature to conclude whether or not a given study is sound. Caveat emptor applies—Let the buyer (or, more appropriately, the public) beware!
What are the attributes of sound science?
“Conducted by qualified personnel” means that the investigators have acquired the necessary expertise, either by formal training or on-the-job experience, to use descriptive and analytical tools appropriately, to design studies that can rule out false or alternative hypotheses, and to communicate the results accurately. They are well acquainted with the subject area and the body of technical information pertaining to it, a fact usually evidenced by a substantial and relevant publication record. “Qualified personnel” does not mean that academic (Ph.D., D.Sc.) or professional (P.E., D.A.B.T.) credentials are either necessary or adequate to ensure that sound research is conducted. At the same time, the great majority of qualified investigators in highly specialized fields today have received advanced post-graduate training.
“Documented methods” enable the original researchers and others to reproduce the results by the same techniques or alternative approaches, to check for possible errors such as inaccurate calculations, and to detect unexplained influences (e.g., contaminated glassware, impure reagents) or systematic biases in the observations (e.g., gender differences within the test population). Well-documented methods are essential to validate data, to provide in-depth review by scientific peers, and to credit the individuals who did the work.
“Leading to verifiable results and conclusions” means that the conclusions are directly supported by the experiment and can be independently validated by others, in other laboratories or observational settings, using the same or other methods. While scientists can comment freely on the significance of their work beyond the scope of their studies, they must be careful to distinguish their speculations from conclusions. Perhaps the most common means of verification used today is peer review of experimental results in manuscripts submitted to professional journals for publication. Scientific advisory boards and expert panels are other common mechanisms used to verify studies or bodies of information.
How should scientific findings be communicated?
Scientific findings should be reported clearly, completely, and accurately, with supporting documentation and data easily available for in-depth review by other researchers in the field. Several guidelines are offered below.
There is a need to know the intended audience and to tailor the content accordingly. Because the same study might appear in a specialized technical journal, a popular science magazine, and a newspaper, the editors of each medium must adapt the amount of detail and background to suit the interests of their readership. The facts, speculations, or inferences of the original authors should be the same in any case, unless it is made clear that an alternative interpretation is being offered. In that case, the same expectations (qualifications of the interpreter, documentation, etc.) would apply to the reviewer or editor, and any new information presented should be subject to the same standards of the scientific method.
Conclusions should be drawn within the boundaries of the data and the scope of the study. For example, observations that a bird population was declining in some localized nesting areas would probably not in themselves support a conclusion that the bird was threatened across its entire range.
The absence of evidence should not be confused with evidence of absence. The failure to find a given or expected outcome, such as a decline in a fish population exposed to a spill of a hazardous material, could reflect the lack of a population-level effect. However, the possibility that the investigational methods used to monitor the population were deficient or insensitive cannot be discounted. An axiom of science is the virtual impossibility of proving something does not exist, even as detection methods continue to improve. Scientific studies at best can demonstrate only what does exist and can report the absence of an outcome under given conditions or probabilities.
Systematic changes in the direction of two or more variables (correlation) should not necessarily be interpreted as one variable causing another to change (causation). For example, increases in global temperatures in recent decades have followed rising atmospheric concentrations of carbon dioxide, but it is still unclear whether or to what extent manmade emissions are the cause.
The relative strength or weakness of available information to support conclusions, conditions where a conclusion may not apply, and alternative interpretations for trends seen in the data should be recognized and discussed. While newspapers compete for eye-catching headlines and late-breaking news stories, conscientious science authors and editors strive to convey the qualifications and factors that contribute to uncertainty, particularly when they relate to conclusions drawn from the study.
How should scientific findings be used in public policy?
Sound science is an important factor in setting policy, as decisions affecting many people clearly should be based on accurate technical information. However, science is by no means the only factor that should be considered in setting most public policies. Social, economic, legal, or political factors are also important, and any of these factors might ultimately determine how a given issue is managed. Science helps to inform decisions, as do cost analyses, legal reviews, and town meetings. Science does not dictate how public-policy decisions should be made.
In order to maintain the credibility of sound science as a decision-support tool, policy decisions should clearly communicate the body of technical information that was considered, the manner in which the information was interpreted in light of the decision, and other factors that may have been considered. Policy-makers applying scientific findings to a given issue or decision are responsible for ensuring that the data are accurate, relevant, and complete.
What are some “watch-outs”?
Statements of absolute certainty
Proof in science is rarely absolute but more often is based on evidence available at the time. Terms such as “always,” “only,” or “never” are rarely used by conscientious communicators. Measures of the confidence of a statistic or observation (e.g., standard deviation, standard error, or confidence limits) help readers weigh the reliability or significance of a result.
Important variables overlooked or ignored
When trying to prove that one factor is related to another, people often overlook things that could shape the outcome of the study. For example, a generalization that “hungry dogs are more dangerous” would overlook important factors such as differences among canine breeds, rearing, and training. Exceptions to the rule and confounding factors in an analysis should be noted.
Unreported or inadequate sample sizes
Studies based on a small group of subjects can lead to false conclusions. When the number of subjects studied or number of experiments repeated is not stated or seems low, it is usually difficult to judge a study’s validity. Well-communicated technical studies give some indication of the sample size (n) of a study, often reported as a number, e.g., n = 97.
Lack of useful standards of reference
Percentages reported without good standards of reference, criteria, or benchmarks can be misleading. An understanding of what is normal or expected is necessary to interpret whether a situation is abnormal. For example, a 100% increase in physical abnormalities in a bird population may sound alarming but may not be unusual or unexpected. A 100% increase in a large sample (e.g., 4 incidences in 100,000 birds when only 2 were expected) may not be statistically significant or ecologically meaningful.
Inferences of cause-effect relationships
Even if two events occur together, one event may not be the cause of the other. Both may be related to another factor that is the true cause. For example, the observation that people wear hats more often when they are wearing gloves does not mean that gloves cause people to wear hats. The wearing of both hats and gloves is likely related to other factors such as cold weather.
Observer bias and vested interests
Even unconsciously, people’s predispositions can shape a study’s results or the way results are interpreted. A source’s record of objectivity is important in determining the validity and relevance of a finding. People with vested interests in an outcome (e.g., financial, political, or social) may have biased interpretations.
Conclusions based on personal stories
People often draw incorrect conclusions from anecdotal evidence, especially when presented in a familiar or appealing manner. Many public-speaking courses emphasize the importance of personalizing a speech or debate to be more convincing. While personal stories and testimonials may add color and enliven a technical study or finding, they do not constitute a means of validation.
Unpublished findings
It usually is not possible to thoroughly assess the technical quality of findings from unpublished studies. Similarly, findings from studies that were never peer-reviewed or published may be sound, but again these results are more difficult to assess. The premature release of unverified data or speculative conclusions to a news-hungry press can be a problem. At times, embarrassed researchers must retract their findings when the work cannot be reproduced or the conclusions are found to be faulty. Not in all cases, however, can unfortunate consequences (e.g., lasting misperceptions, unsupported decisions, compromised reputations) be completely reversed.
About this document
When referring to this text, please use the following citation:
Society of Environmental Toxicology and Chemistry (SETAC). 1999. Sound Science Technical Issue Paper. Pensacola, FL, USA.
Although this document is copyright protected, SETAC encourages you to reproduce and distribute the document in the interest of enhancing communication about the environmental sciences. Two restrictions apply: 1) commercial use, reproduction, sale, or distribution is strictly prohibited, and 2) any copies of this document that you make or distribute must clearly indicate “Copyright Society of Environmental Toxicology and Chemistry (SETAC) 1999.” Contact SETAC before you reproduce or distribute this document under terms other than those allowed here. For more information, contact the SETAC Office nearest you:
1010 North 12th Avenue
Pensacola, FL 32501-3367 USA
T 850 469 1500
F 850 469 9778
setac@setac.org
Av. de la Toison d'Or 67
B-1060 Brussels, Belgium b 6
T 32 2 772 72 81
F 32 2 770 53 83
setaceu@setac.org
Society of Environmental Toxicology and Chemistry
In the 1970’s, no forum existed for interdisciplinary communication among environmental scientists—biologists, chemists, toxicologists—and others interested in environmental issues such as managers and engineers. The Society of Environmental Toxicology and Chemistry (SETAC) was founded in 1979 to fill the void. Based on the growth in membership, annual meeting attendance, and publications, the forum was needed.
Like many other professional societies, SETAC publishes an esteemed scientific journal and convenes an annual meeting replete with state-of-the-science poster and platform presentations. Because of its multidisciplinary approach, however, the scope of the science of SETAC is much broader in concept and application than that of many other societies.
SETAC is concerned about global environmental issues. Its members are committed to good science worldwide, to timely and effective communication of research, and to interactions among professionals so that enhanced knowledge and increased personal exchanges occur. Sister organizations, SETAC Europe (1989), SETAC Asia/Pacific (1997), and SETAC Latin America (1999) have been formed, and the nonprofit SETAC Foundation for Environmental Education was founded in North America in 1990. International acceptance of the SETAC model continues with widespread interest in Russia and Africa.
Download a .pdf version
