One of the pathways by which scientists confirm the validity of a new finding or discovery is by repeating the research that produced it. When a scientific effort fails to independently confirm the computations or results of a previous study, some argue that the observed inconsistency may be an important precursor to new discovery while others fear it may be a symptom of a lack of rigor in science. When a newly reported scientific study has far-reaching implications for science or a major, potential impact on the public, the question of its reliability takes on heightened importance. Concerns over reproducibility and replicability have been expressed in both scientific and popular media.

As these concerns increased in recent years, Congress directed the National Science Foundation to contract with the National Academies of Science, Engineering, and Medicine to undertake a study to assess reproducibility and replicability in scientific and engineering research and to provide findings and recommendations for improving rigor and transparency in research.

The committee appointed by the National Academies to carry out this task included individuals representing a wide range of expertise: methodology and statistics, philosophy of science, science communication, behavioral and social sciences, earth and life sciences, physical sciences, computational science, engineering, academic leadership, journal editors, and industry expertise in quality control. Individuals with expertise pertaining to reproducibility and replicability of research results across a variety of fields were included as well.

This presentation will discuss the committee's approach to the task and its findings, conclusions, and recommendations related to factors that influence reproducibility, sources of replicability, strategies for supporting reproducibility and replicability, and how reproducibility and replicability fit into the broader framework of scientific quality and rigor.

The scientific enterprise operates via a few basic features, including questions, insight, hypotheses, critique, reproducibility, elaboration and revision. All contribute to the process of discovery, none can be taken as the single signature of scientific truth. Discovery science is a dynamic process informed by new observations and continuous refinement of the precision, accuracy, principals and scope of our collective scientific knowledge. History has many examples of significant revisions of previously accepted dogma based on new observations (the earth is flat, matter is infinitely divisible, the stars are fixed). New insights lead to new fundamental principles (energy is conserved, nothing can go faster than light, germs cause disease) that open new opportunities for advancing the scientific frontier and raising the quality of life. Examples of advances of the frontiers of energy science and their implications for reproducibility will be given.

This talk will discuss reproducibility issues encountered during the development of novel catalysts for low temperature fuel cell performance (PEMFC) as replacements of state-of-the-art catalysts that contain Pt-based nanoparticles supported on a high surface area carbon support. Examples across several catalytic systems will be shown, including low platinum-group metal (low-PGM) catalysts with extended surfaces derived from nanowire templates, and PGM-free catalysts based on N-doped carbon with an atomically dispersed transition metal.

Reproducibility issues related to the synthesis of these catalytic materials and their impact on the performance of these catalysts will be reported first following by discussion of challenges in characterization. Specifically, the need for complementary characterization will be highlighted along with issues that arise when materials are characterized by different groups using different techniques. Another set of reproducibility issues arises when conducting characterization of catalysts under in-situ and in-operando conditions.[1] The time constraints imposed by the availability of instrumentation result in datasets that have a limited number of samples, areas per samples and replicate measurements on the same sample.

(1) Dzara, M. J.; Artyushkova, K.; Shulda, S.; Strand, M. B.; Ngo, C.; Crumlin, E. J.; Gennett, T.; Pylypenko, S. Characterization of Complex Interactions at the Gas − Solid Interface with in Situ Spectroscopy : The Case of Nitrogen-Functionalized Carbon. J. Phys. Chem. C2019, 123 (14), 9074–9086.

Charge transfer across heterogeneous interfaces facilitated by redox reactions is the basis of energy storage technology. Capturing the interfacial processes over broad scales both spatially (ranging from angstroms up to 100 nm) and temporally (lasting from fs up to a few minutes) is a major challenge. This is one origin of the existing knowledge gaps in energy storage materials, which impede our ability to predict and control the emergent behaviors at electrochemical interfaces. As part of Joint Center for Energy Storage Research (JCESR) center, we developed a multi-modal in situ characterization tool set based on X-ray absorption, photoelectron and multinuclear NMR spectroscopy in combination with computational modelling that can access a range of the important complex processes. This multimodal approach helps us gain critical insights of the charge transfer process, but also presented unique challenges in data collection, analysis and reproducibility. The multitude of constituents and varying surface chemistry combined with external stimuli (applied potential and temperature) challenges the traditionally conceived time and spatial resolution limitations of the probes. For example, establishing reference systems and base line measurements for electrochemical process where combinatorial constituents react and depend on the charge state is a major challenge in spectroscopic studies and complicates subsequent corroboration with computational analysis. In this talk, we will discuss overcoming these challenges and apply the methods to critically analyzing solid-electrolyte interphase (SEI) evolution in Li-metal based batteries, multivalent ion transport across membranes and chemical stability of redox flow battery electrolytes.