For Yet Better Science

Tools (as well as the research they enable) are not just developed by and for the professional scientific research community, but by a wide range of commercial, academic, nonprofit, and community enterprises operating at a range of scales. Here, we outline 16 tools for science that are causing us to rethink the boundaries of scientific research. Looking across these tools and their individual impact on science and society, we begin to ask questions about their collective impact. How do low-cost tools impact science? Do these tools accelerate scientific progress or expand access, and to what extent? Finally, is the impact, and potential impact, of these tools incremental, or potentially revolutionary?

The panel discusses what inclusive research is, how to conduct it, and what issues and challenges exist about engaging in it. “Inclusive research” has its history as a participatory research method designed to ensure people closest to the issue or problem under study are authentically engaged in the research process rather than simply being “research subjects.” While community-based participatory research has begun to take on greater prominence in the criminal justice realm, such efforts are largely confined to qualitative research inquiries. This panel makes the case that inclusive research can and should apply to a wider array of research questions and methods and that employing it can yield more accurate and policy-relevant evidence. Panelists will also engage in a “myth busting” discussion to address possible challenges to conducting inclusive research and how to overcome them. 

Conducting research with human subjects can be difficult because of limited sample sizes and small empirical effects. We demonstrate that this problem can yield patterns of results that are practically indistinguishable from flipping a coin to determine the direction of treatment effects. We use this idea of random conclusions to establish a baseline for interpreting effect-size estimates, in turn producing more stringent thresholds for hypothesis testing and for statistical-power calculations. An examination of recent meta-analyses in psychology, neuroscience, and medicine confirms that, even if all considered effects are real, results involving small effects are indeed indistinguishable from random conclusions.

The "New Statistics" emphasizes effect sizes, confidence intervals, meta-analysis, and the use of Open Science practices. We present 3 specific ways in which a New Statistics approach can help improve scientific practice: by reducing over-confidence in small samples, by reducing confirmation bias, and by fostering more cautious judgments of consistency. We illustrate these points through consideration of the literature on oxytocin and human trust, a research area that typifies some of the endemic problems that arise with poor statistical practice.  

Collaborations between the academy and governments promise to improve the lives of people, the operations of government, and our understanding of human behavior and public policy. This review shows that the evidence-informed policy movement consists of two main threads: ( a ) an effort to invent new policies using insights from the social and behavioral science consensus about human behavior and institutions and ( b ) an effort to evaluate the success of governmental policies using transparent and high-integrity research designs such as randomized controlled trials. We argue that the problems of each approach may be solved or at least well addressed by teams that combine the two. We also suggest that governmental actors ought to want to learn about  why  a new policy works as much as they want to know  that  the policy works. We envision a future evidence-informed public policy practice that ( a ) involves cross-sector collaborations using the latest theory plus deep contextual knowled

George Washington Carver was an American agricultural scientist and inventor who promoted alternative crops to cotton and methods to prevent soil depletion. He was one of the most prominent black scientists of the early 20th century.

Hilda Bastian considers post-publication commenting and the cultural changes that are needed to better capture this intellectual effort.