The inability to reproduce experimental results is a problem gaining increasing attention. A recent paper (see IBO 6/15/15), reported that about half of the annual spending on preclinical research in the US is on irreproducible studies. Last month, a task force of the American Society of Cell Biology (ASCB) published a white paper on the matter based on survey responses of 869 of its members. Although not statistically valid because of a low response rate, the survey results clearly show irreproducibility to be a common problem—one that affects researchers, scientific organizations and companies producing instruments, software and consumables.

More than 70% of ASCB survey respondents reported the inability to reproduce the results of a published experiment, and almost 200 researchers had been told that their published results could not be reproduced by another lab. The top three factors accounting for the discrepancies, according to respondents who were able to resolve them, were “[i]ncomplete specification of original protocol could not accurately guide the replication attempt” (45%), “[l]ack of appropriate expertise or rigor” (40%) and “[r]eagents (including antibodies, sera, plasmids, etc.) (38%). Based on the survey results, the task force has made recommendations to the ASCB and other organizations, such as better training, implementing specific publishing standards and promoting the development of community-based standards.

According to Mark Winey, PhD, professor, chair of Molecular, Cell and Developmental Biology at the University of Colorado, Boulder, and chair of the ASCB task force, one way the Society is following the report’s recommendations is through member awareness. This has been initiated with an editorial in the ASCB’s newsletter. A piece in the Society’s journal, Molecular Biology of the Cell, is likely next. The ASCB is also encouraging members to develop community-based standards. “We’re looking . . . to see if we can find some energetic, interested, engaged people who would want to try to lead an effort in their particular research community,” Dr. Winey told IBO, indicating that a session at the society’s annual meeting in December could be a way to identify such leaders. He said that the ASCB has also partnered with the Global Biological Standards Institute (GBSI) to develop training modules for postgraduate researchers on best practices for cell authentication.

The report’s recommendations provide opportunities for life science instrument, software and consumables companies. One is to develop standard protocols, reagents and controls for particular types of studies. Some companies already provide recommended protocols and controls with their kits, Dr. Winey noted, but more opportunities exist. Referring to the successful adoption of community-adopted standards in the field of autophagy, he stated, “[T]he advantage to the company, if it works the way it did in the autophagy community, [is] the whole community is going to be using that reagent or that instrument because it’s the standard.”

Reproducibility is more problematic in other cases. As Dr. Winey explained, “It’s when we get into . . . custom reagents, antibodies made to order against an epitope, and what is the standard of proof that they’re really working?” In those instances, and when reagents are used for a variety of assays, new methods may be necessary for validating them. “[A]n antibody or a reagent that’s going to be widely used in all types of assays needs to be validated in each of those assays,” he said. “And that’s sometimes not done.”

He proposed service as another way for firms to join the reproducibility effort. Acknowledging manufacturers’ training and support with the installation of new instruments, Dr. Winey suggested that they think longer term. “[M]aybe we need to look at the user group and how they’re using the instrument a year, year and a half out and make sure it’s all going well and [being used] appropriately.” Companies, he said, could follow up to ensure proper use of instruments for purposes in which users have not been trained. Additionally, he stated, “There might be ways that [companies] could partner with . . . organizations [such as ASCB or GBSI] to make particular training modules that are really focused on people who are new to using a technology—let them know what the issues are around it—and these organizations would have the platform to reach a lot of people.”

Along similar lines, Dr. Winey suggested that manufacturers could support the establishment and maintenance of standards by developing user groups. For example, companies could create online forums to provide protocols, and the community could respond with feedback, including new capabilities and bugs, as they are discovered.