“Let the community decide”? The vision and reality of soundness-only peer review in open-access mega-journals

2.1 Attitudes to peer review

A number of studies have investigated researcher perceptions of peer review, with sometimes contradictory results. Researchers have been found to be generally satisfied with the current system, and appear to highly value the practice (Mulligan et al., 2013; Nicholas et al., 2015; Thelwall et al., 2013; Ware, 2008a, b). In survey results reported by Ware (2008b), an overwhelming majority of respondents believed that peer review was both necessary (93 per cent) and helped scholarly communication (85 per cent). A more recent survey of researchers supported this, finding that peer review is overwhelmingly perceived to be the most important characteristic of a trust-worthy publication (Nicholas et al., 2015). Interestingly the same study also found that negative views of OA journals were often predicated on a belief that such journals do not employ a “proper” peer review process. Studies have commonly reported that researchers believe that peer review helps improve the quality of the papers published, with possible improvements including enhancement of a paper’s discussion, improvements in language and presentation and correction of scientific errors (Bornmann, 2011; Sense About Science, 2009; Taylor & Francis, 2015; Ware, 2008b).

However, many observers believe the peer review system as a whole to be under increasing pressure (Procter et al., 2010). Some go so far as to say it is “near breaking point” (Franklin, 2010; see also Fox and Petchey, 2010). Such tension is often attributed to two interrelated factors: the increasing volume of publications and the globalisation of science. The volume of research produced is growing at a steady but sustained pace, in the region of 3 per cent each year (Ware and Mabe, 2009). It is fuelled in no small part by a significant rise in the number of active researchers in developing countries, such as China and India (Ware, 2011). Evidence suggests, however, that researchers from these countries perform proportionally far fewer reviews than academics in the Global North (Warne, 2016). The result is a growing burden being placed on both reviewers, who are faced with an increasing volume of peer review requests, and publishers, who experience difficulties managing the process and recruiting reviewers.

There is also evidence that researchers are increasingly aware of the shortcomings of peer review (Ware and Mabe, 2015). Only 32 per cent of researchers think that peer review as currently practised is the best system that can be achieved (Sense About Science, 2009), reflecting authors’ awareness of the various limitations and problems identified in the literature, such as bias in the review process (based on a range of factors including gender, nationality and prestige) (Lee et al., 2013), inconsistency among reviewers (Blackburn and Hakel, 2006), the slow and expensive nature of the process (Research Information Network, 2015; Smith, 2006) and the questionable predictive validity of judgements of significance (Bornmann, 2011; Velterop, 2015). Researchers are aware that the importance placed by publishers on the journal impact factor (JIF), which is based on citation counts of articles published in the journal, may provide motivation for editors to publish only those articles that are likely to accrue high numbers of citations (Pulverer, 2013).

In recent years, some innovations have emerged (Tennant et al., 2017). These typically take advantage of technical advances and include open peer review (where reviewer reports and identities are published alongside the article), portable peer review (where the reviewer reports of rejected papers are made available to the editors of other journals), and even machine-based peer review (in which papers are automatically analysed using algorithms to check statistical data, plagiarism, paper structure and grammar). Post-publication peer review, where articles are published after an initial in-house check with more formal peer reviews added later (as, e.g., practised by F1000) represents another alternative model.

Whilst many of these innovations associated with peer review have experimental or early operational implementations, soundness-only peer review is to date perhaps the most widely operationalised. However, it is still little examined in the research literature. Björk and Catani’s (2016) quantitative study on the comparative outcomes of soundness-only peer review is an exception. They compared the citation distributions of OAMJ’s using soundness-only peer review compared with a group of traditional selective titles, finding surprisingly little difference between the two. From this they drew the tentative conclusion that since soundness-only peer review is likely to be quicker and less burdensome than conventional peer review, it may be preferable to the traditional system. Their study was, however, based on a sample of information science “elite” comparator titles, whereas selective journals in the biosciences (the disciplinary area of many OAMJ articles) may have somewhat different citation patterns. The OAMJ sample itself included journals with a range of coverage, including at least two covering the humanities and social sciences, disciplines with different citation conventions. Nevertheless, their study has illustrated the need for further research in this area. To date, however, there have been no published studies on how the soundness-only peer review process actually works in practice and what that means in terms of the vision put forward by proponents. This present study begins to address that gap.

The current study was, nevertheless, carried out in a context in which the conventional model of peer review has proved remarkably resilient. Despite some disquiet about its failings and the development of alternatives to conventional peer review, there has still not been any major challenge in practice to the primacy of the traditional model. Björk and Hedlund (2015) observe that not even the introduction of the web has had a profound impact on the way peer review is conducted for most journals. Even with its shortcomings, conventional peer review is commonly regarded as the best system available and any change to it is often regarded with caution and suspicion (Pinfield, 2015).

2.2 Elements of peer review

Our review of the literature relating to peer review, combined with an appraisal of “instructions for reviewers” available on journal websites, suggests that the traditional peer review process is constituted of four main elements (see Figure 1):

1. Novelty or originality: the way in which the paper makes an intellectual advance contributing in an innovative way to knowledge; this may include innovative methods, new empirical results, novel interpretations or original theoretical insights.

2. Significance or importance: the way the paper adds to the body of knowledge, making an impact by enhancing understanding or practice.

3. Scope or relevance: the anticipated interest of the article to its readership as defined by the journal’s stated subject coverage.

4. Soundness or rigour: sometimes referred to as “technical” or “scientific” soundness, this relates to issues such as methodological precision, coherence and integrity; it includes the quality of the argumentation, relating to the logic of research and the way data are interpreted.

Although conceptually distinct, these criteria may in practice be closely related. An output’s significance may, for example, derive in large part from its novelty, although it is possible to conceive of a novel paper which is not deemed significant or vice versa: a novel piece of research may be trivial in terms of implications and therefore not significant; a replication study or review article may be lacking in originality but nevertheless have significance. Other assessment criteria such as the UK’s Research Excellence Framework (REF) distinguish between novelty and significance, despite their often close relationship. Interestingly, the REF criteria also include rigour alongside originality and significance. Like the other criteria, it applies across different disciplines, and is defined as, “intellectual coherence, methodological precision and analytical power; accuracy and depth of scholarship; awareness of and appropriate engagement with other relevant work” (HEFCE, 2012).

These four elements of peer review are typically applied once a paper has undergone pre-peer review quality checks. These are likely to include plagiarism detection, compliance with file formatting requirements and inclusion of any required supplementary files, including data. The paper will also be initially checked for scope. In addition the content of the paper is likely to be assessed to ensure it meets basic standards of quality of expression and presentation. The latter requirement is in many respects a cross-cutting criterion that relates to all of the above main criteria. Peer reviewers may also comment on clarity of expression and presentation as part of the review process (as indicated on Figure 1 by the partial darker shading in this box). If a paper does not meet a basic threshold from this initial screening, it may be “desk rejected” by the editor of the journal before the main peer review criteria are applied. The extent to which the other peer review criteria are applied by the editor or editorial staff before the paper is sent to reviewers will vary by journal, with some highly selective journals undergoing more in-house scrutiny before circulation to external peer reviewers.

The criteria are all applied before publication in order to determine acceptance of the paper for publication. There are no post-publication components of the formal quality assurance process in the conventional system (they are therefore unshaded in Figure 1). Although judgements of article quality associated with citation count and JIF are inevitably made within the academic community, this is not directly associated with the publication process itself.

OAMJs have generally described themselves as journals for which peer review solely focusses on technical and scientific soundness. A relevant section of the instructions for reviewers of the largest OAMJ reads as follows:

To be published in Scientific Reports, a paper must be scientifically valid and technically sound in methodology and analysis. Manuscripts are not assessed based on their perceived importance, significance or impact; the research community makes such judgements after publication

(Scientific Reports, 2017).

Figure 2 applies these stated aims to our model. Here, and in the OAMJ discourse more generally, the only criterion for pre-publication peer review is technical or scientific soundness (with clarity of expression and presentation also assumed as an initial threshold). Novelty, significance and relevance are excluded from the peer review process, and left for the community to determine and communicate via post-publication quality indicators, such as article-level metrics and comments. Such post-publication indicators are in no way unique to OAMJs but have been viewed as essential to the vision articulated around the concept of soundness-only refereeing.

In the following sections, we analyse and discuss soundness-only peer review and its implications, based on the evidence gathered during the interviews with publishers and senior editors. After an account of our methods, we report definitions of soundness-only peer review suggested by our participants and we analyse arguments they use to justify (or criticise) such an approach. We then discuss some of the operational realities of soundness-only peer review. The elements presented in Figures 1 and 2 are discussed in the context of our findings.

4.1 Defining “soundness-only” peer review

All participants were asked to explain what they understood by the term “soundness-only” peer review. Primarily, it was associated with reviewing the robustness of the approaches taken to assembling, analysing and interpreting the data underlying the study. There was a consensus that this meant evaluating the rigour and ethics of the research, with one interviewee summarising the process as “looking at the basics” (MJ editor 1). Interviewees generally suggested that the specifics of this include assessing whether the correct methodology has been used to address the research questions, that any statistical tests have been used appropriately, and that the research satisfies the necessary ethical requirements. Several participants also stated that the approach particularly scrutinises the conclusions of the paper – “editors and reviewers would judge the work based on whether the conclusions were supported by the data” (MJ publisher 1) – which also serves to identify sweeping claims or attempts to overstate the conclusions by exaggerating the importance and reach of the findings, a practice participants suggested was common in some very competitive disciplines.

It was notable that several participants described the soundness-only approach in more general terms. One suggested that it is “looking at […] [whether] the paper belongs in the published literature somewhere, and if it does, then it can be published in a mega-journal” (MJ publisher 2). Others used potentially more loaded and less clearly defined terms: “if it looks like good science then we are keen to publish it” (MJ editor 2); “there is a minimum acceptable quality standard” (MJ publisher 3). It was notable too that initial descriptions of soundness-only review from a number of interviewees also defined the approach in terms of what it does not do, highlighting the absence of any judgement of significance or importance: “we are not worried about priority or how sexy the science is” (MJ editor 2).

No agreed definition of soundness emerged in the context of the arts and humanities. Several interviewees thought soundness simply could not be transposed to these disciplines. As one publisher put it:

[…] [soundness] doesn’t translate into all fields. I have to say the humanities […] I’d be really hard pressed to really understand […] I mean it can’t translate terribly well

(Non-MJ publisher 1).

Others were more positive, with some acknowledging the important work done by Eve (2014) in attempting to define a set of criteria for soundness in the humanities, essentially emphasising the clarity of the argumentation and the proper use of the sources. This was expanded upon by one participant, who suggested that:

There are ways of pre-filtering on a much lower level for humanities work. For example, […] in humanities you can look for work that is written in an appropriate register that cites all the appropriate literature, that evinces an argument that is self-coherent, that tends to pre-empt criticisms that have been levelled at it. Essentially things that are more to do with the apparatus of a paper than the actual content

(Non-MJ publisher 2).

Mirroring what is found in the literature on OAMJs, and particularly the informal literature (see e.g. Taylor, 2015), it was notable that interviewees used a variety of qualifiers when describing soundness-only peer review or OAMJs. The two most commonly used terms were “objective” peer review and “peer review lite”. “Light touch”, “light weight”, “non-selective”, “impact neutral” peer review, as well as “low peer review barrier”, were also used by some interviewees. The rhetoric here is important. In most cases, it seemed to reflect an internalised value system: advocates of soundness-only peer review were more likely to use the “objective” qualifier, whereas critics tended to use the “light”/“lite” qualifier. Peer review “lite” was often associated with notions of lower-quality and broad acceptance; that such non-selective peer review conveyed a negative connotation. This perhaps explains the importance publishers placed on the choice of the more positive “objective” to describe the peer review process for their OAMJs.

The rhetoric may also reflect the degree of individual engagement with the concept itself. One interviewee (Non-MJ publisher 3), who was extremely vocal in criticising OAMJs, referred to the concept of “sound science” as used by OAMJs as “garbage” since in the interviewee’s view novelty and importance are “entirely part of sound science”.

4.2 Justifying the soundness-only approach

All participants acknowledged that the OAMJ peer review approach represents a significant departure from traditional editorial processes. This is reflected in the fact that several interviewees described in detail the difficulties they faced justifying the soundness-only approach to senior decision makers within their organisation. The removal of significance and impact as evaluation criteria in particular was apparently viewed by some as high risk, and even against the founding values of research excellence.

In response, most interviewees with an OAMJ background suggested that the weight placed on assessment of impact and significance was detrimental to the scholarly communication process and counter-productive to the progress of science, relying as it does on the free flow and exchange of ideas. Thus, the relaxation of the importance/significance constraint on academic publishing was often voiced as one of the main motivations for publishers and editors to engage with a OAMJ venture. They suggested that assessment of impact and significance was the most subjective and unreliable judgement that takes place in the traditional peer review process. One OAMJ editor likened an attempt to predict the impact of an article in five or ten years’ time to “reading a crystal ball”. A recurrent argument regarding the rationale for soundness-only peer review was that it removed this subjective element. Instead it was suggested that the OAMJ approach allowed the entire community to decide what is important. A key word used by some interviewees here was “democratisation”. They argued that the community is best placed to judge which papers are important and significant – more so than editors and reviewers. Such an approach was sometimes framed as a stance against elitism and the gatekeeping attitude that editors are perceived sometimes to exhibit: what an editor finds important may not be necessarily be what an individual researcher might find important. Some interviewees articulated a more extreme characterisation of editorial practices at some journals, suggesting that editors might be motivated to reject work out of self-interest:

I have certainly seen biases in place where a certain editorial viewpoint was being put forth and that contradictory research was basically being held back […] so removing that kind of nonsense from the process is noble

(MJ publisher 4).

This desire to move towards a system encouraging the productive and frictionless exchange of ideas emerged from the interviews. One participant summarised it by saying:

[…] the more journals that can operate this sort of model, the better, because science is not all about exciting, ground-breaking discoveries, a lot of it is about collecting data, verifying that data, providing the ground, the basis for other work

(MJ publisher 5).

Interviewees insisted that research findings which are not regarded as highly significant might still be very useful, providing “incremental steps” for other researchers to take forward. Soundness-only peer review facilitates the publication of just this research. Several participants also described how the approach aligns with the requirements of funders, who are increasingly vocal in their belief that all the results of publically funded research should be published.

A further argument in favour of OAMJ peer review put forward by participants related to the broader scholarly context. Soundness-only peer review was justified by many interviewees as an approach both divorced from and a challenge to prevalent modes of research evaluation. All interviewees recognised that the journal ranking system was a central cultural element of academic publishing, with one describing journal rank and JIF as a prestige marker for authors, an attitude reinforced by the use of such ranking and metrics in academic rewards systems and hiring processes. Many interviewees felt strongly that this system, while deeply ingrained, was fundamentally detrimental to the basic goals of academic research. Several participants noted that such measures are naturally irreconcilable with an OAMJ approach to peer review, since the publication of an article by an OAMJ is explicitly not an indicator of its potential importance. OAMJs, and by extension the articles they publish, cannot therefore be ranked against each other. Some participants felt that valid, sound research ended up not being published because of the filtering applied by traditional journals to improve or maintain their JIF. More specifically, several interviewees noted that within the traditional system little credit is given for the publication of replication or null-results studies, and that those studies usually do not help build a higher Impact Factor. Thus, authors tend not to write such studies, and if they do, publishers tend not to publish them. Soundness-only peer review was presented as a means of counteracting this effect.

It is important to note that not all interviewees were persuaded by the arguments in favour of soundness-only peer review. A key factor for critics of the model was the extent to which the approach simultaneously increased the volume of published articles, while removing a key filtering mechanism for readers. In this respect it was notable that one editor of a well-known OAMJ voiced concerns about the journal’s policy to ignore impact:

[…] they [OAMJ senior management] don’t concern themselves too much with impact, and I think that’s an important issue […] I think the problem then is just the sheer volume, and the fact that they may be watering down the seriousness of some of the content

(MJ editor 3).

This view was echoed and expanded upon by another interviewee, who felt that soundness-only review was failing readers:

I don’t think it is helping […] you know knowledge emerges from synthesised information and it is not doing that […] If you are really catering to readers, then you do care about novelty and quality and importance

(Non-MJ publisher 3).

Several participants also rejected the characterisation of editors as limiting factors in the spread of new ideas. As one put it: “editors are really good people and they want to develop their journals in ways that will help continue to lead the community” (Non-MJ publisher 4). A publisher also suggested that the OAMJ argument around democratisation failed to recognise the fact that those closest to the traditional publication process are themselves members of the academic community:

[…] there is a proletarian view of the community that [mega-journals] are peddling which says that editors and experts and peer reviewers and all the technical editors and the field editors and the manuscript editors who work 30 years in a field and probably know it better than most of the practitioners, that those people aren’t part of the community

(Non-MJ publisher 3).

Finally, the rhetoric of “objective” peer review was called into question by some of the interviewees – even amongst those most supportive of the OAMJ concept. They expounded two main arguments. First, that objectivity – and its corollary, consistency – is achievable only if the discipline encompasses a “well-defined community” with shared values and academic standards. If this is not the case, then it is extremely difficult to achieve consistency and objectivity (MJ publisher 6). Second, that since soundness-only review typically includes an assessment of the validity and reasonableness of the research questions, a certain amount of selectivity, and therefore subjectivity, is inevitable (Non-MJ publisher 4).

5.1 Reviews, decision making and governance

Perhaps the most significant finding to emerge from the interviews was the acknowledgement by many OAMJ publishers and editors that their peer reviewers, despite instructions to the contrary, still regularly factor significance, novelty and impact into their evaluation of articles under review. As one interviewee put it, “the culture of selecting things on the basis of impact is very ingrained in scientists, and it’s very hard sometimes to get them to drop their old habits” (MJ publisher 5). It should be noted that there was an acknowledgement from representatives of the more established OAMJ titles that such reviews were becoming less prevalent, a trend explained by increasing general awareness among researchers of the soundness-only model, and the increasing likelihood that reviewers had reviewed previously for the OAMJ: “If it’s a new reviewer who’s never reviewed for [the mega-journal] before, we’re more likely to get somebody who judges against more standard criteria” (MJ editor 4). There was also a view that even within the strict soundness-only framework, reviewers were reluctant to relinquish the detailed contextual assessment associated with traditional peer review. This was summarised by one publisher: “what [reviewers] are not very good at is just looking at the science that is on the table and evaluating it to see whether it is publishable or not” (MJ publisher 7).

In response to this, interviewees involved in OAMJ production reported the need to provide reviewers with training and guidance materials about what soundness-only peer review is, and what is expected from them. The use of review templates limiting reviewers to scientific soundness and ethical assessment was a common tool deployed to make it easier for reviewers to understand what was expected from them, although it was reported that reviewers would often use comment boxes to circumvent these restrictions.

Given the challenge of changing reviewers’ habits, and the apparent reluctance of some to conduct reviews limited to soundness only, it became clear from the interviews that editors play a crucial role. Several participants described how in practice the model relied on editors to scrutinise referee’s reports and disregard elements relating to significance or novelty. As one editor put it:

[…] when I receive the reviews, I kind of mentally filter out those criticisms of “it’s not an advancement” or “it’s not exciting” and just purely look at the comments and criticisms against the publication criteria

(MJ editor 4).

It is not surprising then that senior OAMJ executives suggested that getting the right editors in place was crucial, as in practice it is they more than reviewers who often implement the soundness-only criterion. Recruiting editors who fully understand and embrace the soundness concept and the founding principles of OAMJs was seen as key to delivering a successful and consistent OAMJ. On this point it is striking to note that a small number of the OAMJ editors we interviewed actually showed some reservation about soundness-only peer review, and in two cases professed total ignorance that the journal they edited implemented such a policy!

The distributed editorial model generally adopted by OAMJs, with editors responsible for the review and decision processes for individual papers without any broader strategic role in the journal, was perceived to exacerbate governance challenges. In particular the structure of the OAMJ was reported to create risks around consistency of decision making: the flatter the structure of the OAMJ, the more control is devolved to the hundreds or even thousands of editors. Often supported by algorithmic matching of editors with new submissions, such a model was reported as essential to a journal’s ability to scale, but problematic from a quality assurance perspective. One interviewee – who is active in the scholarly communication debate and highly critical of OAMJs – was particularly vocal about the pitfalls of the OAMJ editorial structure. This participant suggested the system represented:

[…] a federated approach to editorial quality not a guided accountable approach […] there is nobody at the head who is going to get fired if something goes wrong

(Non-MJ publisher 3).

This argument was, however, countered by a OAMJ publisher, who reported that, “there is sacking, quite regular sacking of editors” (MJ publisher 1).

Most representatives of OAMJs we interviewed described quality control mechanisms in place to ensure editorial decisions were consistent, and in-line with the OAMJs editorial policies. It was notable that the nature of these mechanisms was closely linked to the size of the journal. Several interviewees described processes in place during the formative stages of their journal that were broadly similar to those found within traditional journals, with an editor-in-chief or section editor personally reviewing all accepted submissions prior to publication. Participants explained that such practices were, of course, unsustainable for journals publishing thousands of articles per year, and instead large OAMJs were found to rely on post hoc measures. Examples of auditing processes included the review of correspondence between editors and authors, and the calculation and monitoring of acceptance rates. One interviewee described a programme whereby editors with unusually high or low acceptance rates were informed of their figures, along with those of their peers, and subjected to increased levels of oversight. It is perhaps not surprising that some participants reported that such measures could lead to tensions between in-house staff and academic editors.

Some journals were said to have adopted a more innovative approach centred on the idea of consultation, often via discussion fora, in particular for editors to seek advice from peers and share best practice. It should be noted that not all OAMJs appear to offer such services, and some OAMJ editors felt that the training they received was primarily focussed on managing the software and technical elements of the review process rather than emphasising the selection criteria. Overall, despite OAMJ representatives’ faith in their operational processes to achieve consistency, there was some recognition that this was a particularly difficult task, and it was reported by several participants that acceptance rates could vary significantly between editors.

5.2 Objectivity, novelty and triviality: the OAMJ quandary

OAMJs were in part envisioned as a means of removing novelty and significance constraints (as they were seen) from academic publishing. Anecdotal evidence and careful examination of selection criteria published on OAMJ websites, however, suggested that this might not always be clear. There was a sense that the demarcation lines between sound science and new/significant science were blurred. Clarification on this issue was sought in the interviews with senior editors, board members and OAMJ publishers.

An inextricable and complex relationship between soundness, novelty and triviality emerged from the interviews. It was acknowledged that in the strictest interpretation, a OAMJ should publish anything deemed to be technically sound, regardless of any lack of novelty, and no matter how trivial. Some participants believed their OAMJs adhered to this interpretation:

[…] we do mean sound science as in you’re a scientist, you work somewhere, you’ve gone through the proper process – and that’s all it means […] you can’t say “sound science plus”. You’re then a different kind of journal, you’re then a selective journal

(MJ publisher 8).

The overall picture, however, was more complex. Most participants agreed that there was a general need for the publication of more replication and null studies – particularly in the medical and life sciences – and the OAMJ approach was designed in part to facilitate this. Early players in the OAMJ landscape had recognised that need and overtly welcomed such papers. However, as one interviewee explained:

[…] the mega-journal works when people are honest in their intentions and that the reason they are doing it is because want to share their discoveries and as soon as it becomes cynical and it becomes about getting a publication, then the model falls down I think

(MJ publisher 9).

Several participants noted that PLOS ONE originally operated with a strict policy to publish anything sound, until it became clear that authors were submitting unexpectedly large numbers of what were described as “replications of meta-analyses” – papers that essentially added nothing to the scholarly discourse, and were written only to benefit the publication record of the authors. The influx of such papers was seen to have damaged the journal, particularly in terms of its credibility with authors. It was in response to this phenomenon that PLOS introduced a requirement for papers to be “worthy of inclusion in the published scientific record”. As one interviewee noted:

[…] the philosophy of the founders of PLOS ONE, it really was publish everything including trivia, it really was about that and what happens is in practice it is very hard to implement

(MJ publisher 1).

The changing of the PLOS ONE policy, while apparently relatively minor, nonetheless represents an important concession that not all sound science should be published.

Numerous other interviewees provided hypothetical examples of sound but trivial science unworthy of publication: calculating the boiling point of water at sea level; or counting the number of corals encountered on a walk across a reef. It was clear though that many interviewees were somewhat uncomfortable with the implications of this, not least because many had previously highlighted the OAMJ philosophy that research does not need to be new or exciting to be published. The response of two interviewees, when asked directly about whether papers were required to be novel or non-trivial, illustrate the issues and apparent contradictions publishers face reconciling a soundness-only policy with the realities of academic publishing:

You could end up with a really trivial result that actually isn’t worth publishing, but technically you’re not able to reject it if you’re on objective peer review. So there are some things that we need to work on there

(MJ publisher 5).

[…] although there is not a test for novelty in that way, because everything that actually passes the peer review hurdle can be published, but we’re not necessarily […] but if we do have an editor in place, if he does think that the research is just truly useless, even if it’s passed the peer review hurdle but it’s reproducing something that we’ve done a thousand of times before, we still won’t accept it, because we do want to be […] there would need to be some kind of quality threshold

(MJ publisher 10).

It was recognised that “some kind of quality threshold” might easily be interpreted as a subjective judgement beyond just the soundness of the science. As one interviewee asked: “What is so completely trivial that it is not science? What is that?” (MJ publisher 1). A range of suggestions were offered by interviewees. One suggested the test was whether the work was “useful in some way to somebody” (MJ publisher 11), while another believed that the article should be “an original contribution […] adding to the core of scientific knowledge in any way” (MJ publisher 12). One participant set the bar higher, stating that replication and null result studies were rejected unless they “show that some major idea was wrong” (MJ editor 5), a significant departure from the stated aims of most OAMJs. In the case of non-STEM disciplines, novelty appeared to be an even more difficult issue: the body of knowledge cannot always be tested and often consists of paradigms for which soundness makes little sense.

5.3 Quality and the pressure of the JIF

The JIF and rankings based on it, have been one of the most important drivers in academic publishing over the last few decades, and we asked participants for their views on the apparent tension between OAMJs’ vision, and the pressure that JIF is putting on the publishing sector.

A majority of OAMJ publishers recognised this apparent tension, but were also quick in their disavowal of the JIF as a meaningful measure. In support of such claims, some mentioned that they were – or were about to become – signatories of the San Francisco Declaration on Research Assessment (DORA) (www.ascb.org/dora/) agreement. One of DORA’s aims is to combat the implicit association that is made between JIF and the quality of individual papers. Advocates of DORA believe this perceived correlation between JIF and quality is unfounded and may mislead the appraisal of scientific research, be it in hiring and reward mechanisms, funding selection or simply in research-quality performance audits. It was recognised that arguments like this underpin the initial development of the OAMJ concept.

Many participants noted that the OAMJ is not conducive to achieving and maintaining a high JIF. An obvious point made here was that by removing any assessment of likely impact from the selection of articles, OAMJs were likely to publish large numbers of articles accruing few citations: “almost by definition an objective peer review journal is never going to be better than mediocre” (Non-MJ editor 1). Several other factors were perceived to limit OAMJ Impact Factors. The OAMJ goal of publishing null-result and replication studies was mentioned, since they do not usually attract large numbers of citations. Similarly, certain subjects have different citation patterns, which can affect the JIF. With their broad subject scope, OAMJs are likely to publish papers in disciplines and sub-disciplines with low overall citation rates. This relates to particularly to OAMJs that publish research in the social sciences and humanities (which have traditionally much lower citation rates than the sciences) alongside articles in the science, technology and medical areas. Finally, it was noted that fast growing, large-volume journals tend to receive lower Impact Factors, since the denominator of the JIF formula rises disproportionally to citations.

While on the one hand OAMJs can be considered standard-bearers for a new scholarly communication paradigm in which the role of the JIF is marginalised, there was also a clear recognition from participants that OAMJs can and do benefit from achieving and maintaining a respectable Impact Factor. It was widely noted that several OAMJs experienced a surge in submissions following the award of a JIF, and several publishers reported a strong belief that JIF was a primary motivating factor for authors to submit to a journal. OAMJs are therefore subjected to what one participant termed “contradictory pressures” (MJ editor 4). This was articulated clearly by one publisher:

The impact factor is still the metric that is most commonly used […] And so we certainly would like the impact factor to go up and of course there is a little bit of tension between wanting an impact factor go up but claiming that we don’t care about the importance of the article, just the soundness

(MJ editor 5).

Several interviewees noted the relatively high Impact Factors of some OAMJs, and viewed these with suspicion. The implication was that such journals had expanded the criteria of their peer review beyond simply soundness. As one participant suggested, “there must still be an element of gatekeeping going on to enable these impact factors to be so high. I think there’s a lot more rejection than is made out” (MJ publisher 13). That this might represent a deliberate strategy on the part of the publisher was strongly disputed by representatives of the journals in question, with one stating that they were expecting to see their impact factor drop. Academic editors of the journals agreed that they had never been subjected to overt influence: “I’ve never been encouraged one way or another, when I’m looking at a paper, to think whether it’s going to help an impact factor” (MJ editor 4). However, others believed that the process might occur almost “accidentally”, with OAMJ staff previously unconcerned by their Impact Factor “cutting off the flow of some papers” in order to maintain an unexpectedly high JIF (Non-MJ publisher 4). It was also felt that academic editors could be influenced by a journal’s Impact Factor without any explicit instruction from the publisher to do so:

As the impact factor goes up, we will see instances in which the peer reviewers say everything is fine, and then the Editor makes a decision saying “I want to reject this because it’s not interesting enough”

(MJ publisher 12).

There was also a suggestion from one publisher that OAMJs in general, while perhaps not expanding the scope of the peer review, were instead filtering out submissions based on their type:

Most mega-journals don’t publish negative results, because I think it brings citations down. Remember it’s still a business model. I’m giving you the very honest, frank publisher view

(MJ publisher 8).

A recurring argument from OAMJ advocates was that the JIF was not a reliable indicator of the quality of individual articles. For some, quality was directly and clearly linked to certain levels of novelty and significance; for others, quality referred to a more fundamental characteristic of the research described in the article. There was recognition that there were some outstanding papers in OAMJs, and it was suggested that the vast increase in research outputs meant that these papers were no longer making it into traditional journals, owing to the fact that they were still constrained by the number of papers they can accept. Some interviewees commented that the negative views of OAMJs held by some publishers and authors, who were often felt to perceive them as lower-quality publications or “dumping grounds”, were directly related to the OAMJ peer review policy. According to some interviewees, this “dumping ground” perception of OAMJs seemed to be particularly prevalent in certain fields, for example, biomedical sciences, perhaps owing to the influx of papers of variable quality in these disciplines.

In response, interviewees often insisted that low selectivity does not necessarily mean low quality. A recurring argument was that the definition of quality had been distorted, and had become too closely linked to the JIF of the publishing journal. One interviewee argued that:

[…] the concept of a dumping ground is again I think predicated on the wrong definition of quality. An article is not poor quality simply because the findings are less exciting than some other article. I think we need to get away from this what I call almost celebrity scientist culture

(MJ publisher 5).

Most interviewees refuted the dumping ground claim, but did acknowledge the extreme variability of quality in papers published in OAMJs. Far from seeing this as an issue, it was seen as the result of OAMJs being conceived as much as repositories of papers as journals in the traditional sense. Given the discovery tools to which scholars have access, it was felt acceptable to have papers at both ends of the quality spectrum, from excellent to marginal quality, from highly cited to not cited at all, in one single place.

5.4 The limitations of post-publication indicators

As has been discussed, post-publication article level and altmetrics are often considered integral to the OAMJ concept. When asked how effective such indicators are in practice, however, our interviewees were circumspect. Some acknowledged that traditional post-publication measures such as citations or article downloads can provide useful feedback for readers regarding the potential importance or quality of an article, albeit with some limitations; citations, it was noted, take time to accrue, and therefore provide little in the way of immediate information. Several interviewees also acknowledged that commenting functionality is rarely used.

Much of the discussion of post-publication indicators centred on the use of altmetrics. There was general agreement that such measures could be useful for authors and journals as a means of gauging levels of social interest in an article in essentially real time. Views were much more mixed though on the extent to which altmetrics served as indicators of an article’s importance or significance. One participant suggested that it was too soon to determine how best altmetrics might be used:

I think altmetrics are really important, but I am not entirely sure we know what they mean. So it’s great to have a measure of how an article is received, but I’m not sure we know how to interpret those altmetrics scores properly yet

(MJ publisher 12).

This was echoed by other interviewees, who felt that there was not yet enough evidence to form firm opinions on the validity and usefulness of the measures. A number of others, however, were able to point to perceived issues with altmetrics. Typically these related to the potential for funny, quirky or controversial articles to achieve high altmetrics scores, a phenomenon participants linked to the measure’s lack of a qualitative element. As one interviewee put it:

You can get a massive altmetrics score if you’ve got a funny article. They’re indicators. I think there is a need for a qualitative assessment as well as quantitative how many views and downloads. Useful, but where do they come from, why is it happening?

(MJ publisher 13).

A number of participants felt that it was this factor that inhibited altmetrics’ potential usefulness as an indicator of an article’s importance, suggesting that “quantity is no predictor of real significance and impact” (Non-MJ publisher 4). Indeed some were relatively scathing about arguments for using altmetrics in such a way: “altmetrics are used in a very silly way which is that somehow magically readers are going to be able to psychically pick out good content from bad content” (MJ publisher 3).

Several interviewees felt that no metric would ever replace the need for researchers to read articles they were seeking to evaluate. Others felt that better tools to support this evaluation process were possible, and that the publishing industry could do more to make use of technologies being developed in other industries. One participant, while agreeing with this, noted that:

Building those sorts of tools that enable researchers to get connected to the right bits of content, published however, disseminated however is actually incredibly challenging and difficult and expensive to do

(MJ publisher 3).