Summary of CMS Seminar Club presentation on Friday, May 26, 2023.
Title: Transparent Publishing and Open Science – how to share reproducible data
Speaker: Dr. Bernd Pulverer, Head of EMBO Press and Chief Editor of EMBO Reports, Heidelberg, Germany
On Friday, May 26, Dr. Pulverer gave a presentation at Fujita Health University. He told us about scientific publishing in general and about the possibilities of publishing in EMBO Press journals.
Recording: For members of Fujita University, a recording of the meeting (without the discussion part) will be available at our Manabi system. Unfortunately, we cannot open the recording for a wider audience.
There were 30 participants who enjoyed the meeting, and I only heard positive reactions. Already during the meeting, a visitor stated that he had learned many things from Dr. Pulverer’s presentation that he will be able to use in his function as editor-in-chief of journals. Also, Prof. David Alexander wrote me afterward: “Another nice seminar. I liked his point about how reviewers should not just try to find reasons for rejecting a submitted manuscript, but rather the reviewer should try to help the authors “fix” the manuscript so that it is suitable for publication. This is something I also try to do.” The enthusiasm about Dr. Pulverer’s seminar was also expressed in the answer of those audience members whom I asked whether they would like to meet Dr. Pulverer later this year individually—for discussing their work and its possible submission to an EMBO Press journal—all saying that they would like that. So, we are now trying to set a day for his visit, probably in December.
Personally, I was very impressed by seeing how approachable Dr. Pulverer—nonetheless one of the leaders in scientific publication—was, and how many initiatives EMBO Press developed to improve the quality of publishing and the experience for authors.
Dr. Pulverer emphasized that the journal form of scientific publishing still greatly contributes to scientific progress, but that the system is under stress and that all scientists—authors and editors alike—should work together to protect the quality of the science enterprise.
THE CONTENTS OF THE PRESENTATION
Below, I shortly summarize most but not all of Dr. Pulverer’s presentation.
History of Scientific Publishing in Journals
As Isaac Newton stated in 1676, “If I have seen further, it is by standing on the shoulders of giants.” Indeed, scientists need to communicate their work. In that spirit, the first two scientific journals were launched in 1665, Journal des Sçavans (English: Journal of Scholars) in Paris on January 5, and Philosophical Transactions by the Royal Society in London on March 6 (Fig. 1A). Already in 1751, the latter of these two journals formulated rules for how a selection should be made among the increasing number of submissions (the following sentence uses the writing style of those days): “But the Society being of late years greatly inlarged, and their communications more numerous, it was thought adviseable, that a Committee of their Members should be appointed to reconsider the papers read before them, and select out of them such, as they should judge most proper for publication in the future Transactions” (Fig. 1B). Quite charming (for lack of better words) in this sentence is the “read before them”, as these apparently were events in which one person was reading a handwritten text to multiple people.
Dr. Pulverer described that when he started working at Nature in London in the 1990s, there were still colleagues who remembered how earlier that century, when they would receive a newly submitted paper, they would sometimes walk over across the street to the Royal Society to ask scientists who happened to be at the bar for comments.
Only in the 1950s, a formal peer review process was added to the already existing selection by editors/members. Fig. 2 lists criteria that, according to Dr. Pulverer, quality peer review should fulfill: Important is that reviewers should be unbiased and try to have constructive criticism, and that authors should get a chance to defend themselves.
EMBO Press, as do other journals to varying degrees, also makes its selection among submitted studies in consideration of the conceptual advance/novelty. Dr. Pulverer argues that this helps readers to identify the most interesting studies among an ever-increasing number of new studies, and the level of novelty contributes to the hierarchy (reputation/authority) of publication venues (journals). Nevertheless, to promote reliable science, in their journal EMBO Reports, in exceptional cases they also allow publication of confirmatory data (see below).
Scientific publishing has become a very dominant parameter in the science enterprise because it is highly valorized in research assessment. This has been stated very bluntly by the National Academies Press in 2003, which declared: “The publication of scientific information is intended to move science forward. More specifically, the act of publishing is a quid pro quo in which authors receive credit and acknowledgment in exchange for disclosure of their scientific findings.”
Especially because publishing in journals can be quantified, it is commonly used by scientific institutions (e.g., employers and funding agents) as a major determining factor in decision-making. Often used for such evaluation are the Journal Impact Factors (JIF)—based on the average number of citations of articles published in the last two years—of the journals in which an article is published. Dr. Pulverer considers the strong focus on JIF as “extremely toxic because the number is massively misleading” and believes that in combination with the “publish or perish” pressure it leads to gaming of the system by journals and authors alike. For example, a journal can increase its impact factor by publishing more reviews, because those generate many more citations (Fig. 3), and likewise many researchers (who often get evaluated for the citation numbers of their publications) tend to write more reviews than is helpful for the field.
In 2013, this realization brought about the San Francisco Declaration on Research Assessment (DORA) to improve the ways in which researchers and the outputs of scholarly research are evaluated. Institutes and individuals can sign this declaration. Dr. Pulverer is one of the original signees of this declaration, and during his presentation advocated several improvements that could be made in evaluating scientists (please read the DORA text for more explanation).
In short, journals and their panels of experts provide enrichment of articles, by selecting and improving them, and by adding context and analysis (such as added reviews), search technology (see below), and a pleasant layout. Meanwhile, the journal publishing system is under stress by being gamed and as a community of scientists we should try to protect a system that is important to us.
EMBO Press and its journals; a not-for-profit organization with ideals about scientific publishing
EMBO is a global community, although localized and originated in Europe, and is a not-for-profit organization. They hope to intensify their connections with Japan. Their only purpose is to support scientific excellence. The latter results in a high selectivity for content in their journals, whose origin and also present major focus are in the field of molecular biology/medicine.
Only ~25% of the articles submitted to EMBO Press journals are sent to reviewers. Of those, only ~50% are encouraged to continue the publication process by making changes for a resubmission, but almost all of those are eventually published. Thus, in total, just over 10% of all submitted papers are published.
When explaining what EMBO Press looks for in articles, beyond quality and novelty, a first characteristic that Dr. Pulverer mentioned was “Depth (Mechanism),” which in the tradition of EMBO usually means the degree to which molecular pathways have been clarified. In my personal experience, EMBO Press articles are unusually solid and therefore prestigious in this matter indeed. Something else preferred by EMBO Press is “physiological relevance,” as opposed to studies that only are theoretical or artificial, although in EMBO Reports such studies may get published (see below). In my opinion, it is important to understand which journal is a good match for a study, otherwise a lot of time and energy can be wasted in the submission process.
The different journals published by EMBO Press, shown in Fig. 4, are:
• The EMBO Journal. This is the flagship journal of EMBO Press, and its impact factor is 13.8.
• EMBO Reports. The impact factor is 8.8.
The EMBO Journal and EMBO Reports both publish the majority of their articles open access and have a similar broad scope in molecular, cellular & developmental biology. However, EMBO Reports publishes shorter stories that often are not elucidated in as much mechanistic detail yet. EMBO Reports also has a Science & Society category where researchers can describe science from a social perspective (examples in Fig. 5) and an Exploratory report category for more speculative articles. Recent developments for the improvement of science, which resonated well with our webinar audience, are that EMBO Reports also publishes 1st confirmation studies (which are the first to confirm an important new finding by others) and Null data & refuting studies (which say that results could not be found/confirmed).
• EMBO Molecular Medicine. This is a fully open access journal at the interface between basic research and biomedicine. The impact factor is 14.0.
• Molecular Systems Biology. This is an open access journal dedicated to systems biology, synthetic biology, systems medicine, and genomics. The impact factor is 12.7.
• Life Science Alliance is dedicated to “the full spectrum of life science and biomedical research” and was launched in 2019 by EMBO Press together with Rockefeller University Press (the publisher of for example Journal of Experimental Medicine) and Cold Spring Harbor Laboratory Press (the publisher of for example Genes and Development). The impact factor is 5.8.
Among its journals, EMBO Reports appears to be the journal in which EMBO Press is the most communicative, proactive, and creative in trying to establish new ways to improve scientific publishing.
Open Science, and not just Open Access publication
Open Access means that other researchers can use and reuse your information (e.g., figures), as long as the source is properly mentioned. This is generally paid for by the authors, which leads to the inequity that some authors in poorer countries have no access to this publication form. However, the trend in the USA and Europe is that governmental organizations mandate Open Access publication of studies that they fund.
Dr. Pulverer stresses that Open Access of articles is only one aspect of what he calls “Open Science.” That Open Science can incredibly speed up scientific progress was already shown more than 20 years ago when the various groups sequencing (different parts of) the human genome shared their new data publicly in real time. Dr. Pulverer believes in the synergy between Open Science (e.g., preprint systems) and journal publishing.
Described below and shown in Fig. 7 are various aspects of Open Science that EMBO Press tries to improve. Because all these things cost extra money, many other publishers/journals are not making similar commitments. However, Dr. Pulverer believes these commitments are necessary to improve the reproducibility of studies, prohibit the reviewing system from collapsing, and allow the analysis of the ever increasing data by software (artificial intelligence).
Data storage
EMBO Press wants the data associated with an article to be as complete as possible. Therefore, for each article they create a data hub for which the readable (printed) article is almost like an avatar. The data are stored and presented in certain ways so that they can be found and analyzed by computers, including the possible addressing of questions not asked by the data authors. The EMBO initiative SourceData is a tool allowing biomedical scientists to share figures and the underlying source data in a way that is machine-readable. SourceData, together with the Swiss Institute of Bioinformatics, developed SDash for helping with this (SDash has been described as the SourceData “dashboard”).
Full descriptions of Methods
A large reason for the “reproducibility crisis” in science (see also below) is that commonly the descriptions of materials and methods are less than minimal (many of you will be aware of that). Therefore, EMBO Press encourages the posting of full descriptions of materials and protocols, for example by using the online storage facility protocols.io.
High quality reviews are precious
Dr. Pulverer stresses that the reviewing system is not broken, and that on average reviewers spend seven (unpaid!) hours on an article. At EMBO Press the quality of the reviewing process is additionally guaranteed by the involvement of in-house editors. However, in science, the reviewing system is under stress, because more and more articles are being submitted each year and many articles are re-submitted to other journals multiple times.
Therefore, for optimizing the reviewing process, EMBO Press has developed Transparent Process. For a detailed description, please see the link, but important points are that the reviewers’ comments are published together with the article and that authors and their articles are quite well treated and protected during the whole process. In my personal opinion, such good treatment is only possible because 75% of the (generally poorer) articles submitted to EMBO Press are not even sent to reviewers, so the remaining 25% are more deserving of the effort.
After the reviews have been obtained, EMBO Press allows submission of the articles together with the reviews to other journals (and can even make suggestions for that), or to publish them online. EMBO Press also supports the initiative Review Commons (Fig. 7) in which authors can get reviewer’s comments for their article without, initially, submitting it to a journal. Then, together with these reviews, they can submit their article to one of the journals supporting this initiative, or they may even choose to just post it online themselves (temporarily, as a reviewed preprint, or even indefinitely). In this development, interactions with a journal become a chosen added value rather than an automatic necessity.
Research integrity/reproducibility
According to Dr. Pulverer, problems with research integrity and reproducibility are largely caused by the habit of scientist to use figures (or even materials and methods descriptions) to support a narrative and not as a source of data. He is probably right that demanding better quality of submitted data, and how figures represent those data, will significantly improve reproducibility and reduce the options for researchers to cheat.
Dr. Pulverer also described how much time and effort EMBO Press has to spend on research integrity issues, and he showed a number of examples of cheating versus more innocent beautifications. EMBO Press has a set of measures for this, which nowadays includes that all figures have their own authors and that figures can be retracted individually (while the rest of the paper is not retracted).
