For Yet Better Science

Conducting research with human subjects can be difficult because of limited sample sizes and small empirical effects. We demonstrate that this problem can yield patterns of results that are practically indistinguishable from flipping a coin to determine the direction of treatment effects. We use this idea of random conclusions to establish a baseline for interpreting effect-size estimates, in turn producing more stringent thresholds for hypothesis testing and for statistical-power calculations. An examination of recent meta-analyses in psychology, neuroscience, and medicine confirms that, even if all considered effects are real, results involving small effects are indeed indistinguishable from random conclusions.

From leading-edge discoveries in astronomy, chemistry, and medicine, to inventing revolutionary devices, these women have made an indelible impact on our understanding of the world.  

The "New Statistics" emphasizes effect sizes, confidence intervals, meta-analysis, and the use of Open Science practices. We present 3 specific ways in which a New Statistics approach can help improve scientific practice: by reducing over-confidence in small samples, by reducing confirmation bias, and by fostering more cautious judgments of consistency. We illustrate these points through consideration of the literature on oxytocin and human trust, a research area that typifies some of the endemic problems that arise with poor statistical practice.  

Tools (as well as the research they enable) are not just developed by and for the professional scientific research community, but by a wide range of commercial, academic, nonprofit, and community enterprises operating at a range of scales. Here, we outline 16 tools for science that are causing us to rethink the boundaries of scientific research. Looking across these tools and their individual impact on science and society, we begin to ask questions about their collective impact. How do low-cost tools impact science? Do these tools accelerate scientific progress or expand access, and to what extent? Finally, is the impact, and potential impact, of these tools incremental, or potentially revolutionary?

The panel discusses what inclusive research is, how to conduct it, and what issues and challenges exist about engaging in it. “Inclusive research” has its history as a participatory research method designed to ensure people closest to the issue or problem under study are authentically engaged in the research process rather than simply being “research subjects.” While community-based participatory research has begun to take on greater prominence in the criminal justice realm, such efforts are largely confined to qualitative research inquiries. This panel makes the case that inclusive research can and should apply to a wider array of research questions and methods and that employing it can yield more accurate and policy-relevant evidence. Panelists will also engage in a “myth busting” discussion to address possible challenges to conducting inclusive research and how to overcome them. 

Conducting research with human subjects can be difficult because of limited sample sizes and small empirical effects. We demonstrate that this problem can yield patterns of results that are practically indistinguishable from flipping a coin to determine the direction of treatment effects. We use this idea of random conclusions to establish a baseline for interpreting effect-size estimates, in turn producing more stringent thresholds for hypothesis testing and for statistical-power calculations. An examination of recent meta-analyses in psychology, neuroscience, and medicine confirms that, even if all considered effects are real, results involving small effects are indeed indistinguishable from random conclusions.

The "New Statistics" emphasizes effect sizes, confidence intervals, meta-analysis, and the use of Open Science practices. We present 3 specific ways in which a New Statistics approach can help improve scientific practice: by reducing over-confidence in small samples, by reducing confirmation bias, and by fostering more cautious judgments of consistency. We illustrate these points through consideration of the literature on oxytocin and human trust, a research area that typifies some of the endemic problems that arise with poor statistical practice.