An update on progress with recent projects
As research practice and communication moves increasingly online, the scholarly record is expanding in terms of content and scope. In addition to the traditional text-based outcomes of journal articles and books it now includes data sets, software and materials generated in the process of research. A complete research record helps to ensure the integrity of the research process and – as recognised in the research integrity concordat – funders, researchers and those who employ researchers share the responsibility to support the highest standards of rigour and integrity.
In recent years there have been growing concerns about the completeness of the scholarly record which affects the reproducibility of research. One way to find out if new research findings are reliable is to repeat the original research that produced them. If this fails, further questions need to be asked about the validity of the original research.
At Jisc we explored a couple of aspects around these concerns to see what role there might be for tools to improve the reproducibility of research.
In partnership with the University of Edinburgh we explored how mining data from articles on animal-based research helps to detect which factors influencing reproducibility (such as sample size calculation, control group allocation, compliance with ARRIVE guidelines for animal research) are reported in these articles. If researchers, funders, publishers and institutions had a dashboard helping them to see where these key factors are not being declared they could design interventions, such as awareness raising and training, to help improve the situation. Together with our partners we will now consider next steps to further develop the dashboard, also to ensure that it is used in a responsible way. We are also considering how the participatory design and development approach of Jisc analytics labs, which led this work, might be useful for a wide range of other questions of research policy and practice.
Sharing primary data collected during a research study supports reproducibility but, in many cases, the implementation of data sharing maybe less effective than apparent. Together with the University of Wolverhampton and representatives from the UK Reproducibility Network (UKRN) we explored the extent of data sharing of summary statistics of primary human-genome wide association studies (GWAS) as an example of data sharing in favourable circumstances and whether such checks can be automated. Articles sharing GWAS summary statistics usually reported this in a data availability statement within the article. We found that out of 330 articles classified as GWAS in PubMed only 10.8 % reported sharing GWAS summary statistics in some form increasing substantially from 4.3% in 2010 to 16.8% in 2017. Information about whether data was shared can be extracted from data availability statements, but it is more problematic to identify the exact nature of the shared data. Data availability statements are vague about what is shared and there is no standard or policy by journals in a specific field regarding what should be included. Descriptions of the exact nature of the data available would help not only automation but also researchers to find relevant data for a new study. We are now exploring in what contexts automated approaches to extract data availablity statemets could be useful and how representative the GWAS example is in a community meeting.
The completeness of research publications is also a necessary condition for effective peer review, both for journals and for funders so that they can determine the robustness of research findings or proposals. While peer review continues to play a pivotal role in validating research results it has come under strain due to a number of factors. It is perceived as slow and inefficient, seen as a potential source for bias and there have been increasing retractions of peer reviewed papers. Working with the University of Wolverhampton we’ve explored how technology could be used to support reviewers and editors. As part of the Jisc open metrics lab we have experimented with sentiment analysis of F1000 open peer review reports to build a tool which can detect positive and negative evaluations in these reports.
Transparency may influence the extent or nature of judgement biases, and there is now growing evidence about the implications of various open peer review models. The project also tested if national affiliations of article authors and referees reading previously published open peer review reports have an influence on peer review judgements. The evidence from this work could help with building confidence in the open peer review process. We also discussed opportunities and ethical challenges around the use of AI technologies in peer review in a briefing paper. We are currently reviewing the outputs from this project and will publish them shortly on this blog.
Universities have an interest in most forms of peer review as they are influenced by its outcomes. Some universities are directly involved in peer review to prepare their submissions for the Research Excellence Framework (REF). We have been working with the University of Bristol on novel ways to support institutions preparing for funder research assessment. The aim of this work was to develop and evaluate a prediction market tool that universities can use to rank outputs for potential REF submissions as part of their internal REF planning. The idea behind this is that the probability of events can be measured in terms of the bets people are willing to make. Prediction markets have been used to forecast for example elections or film revenues and are used here to predict ratings of research outputs. We expect this approach may be most valuable as part of an evaluation pipeline, including machine learning, prediction markets and close reading by reviewers and, with Bristol, we are continuing pilots to establish what factors influence the best composition of this pipeline in different contexts.
Citations are another aspect of the scholarly record and metrics derived from citation data continue to play a strong role in research assessment at least for journal articles in most STEM disciplines. It is good to see that there has been a world-wide push back against inappropriate use of metrics to evaluate research such as the use of the Journal Impact factor (JIF). In the UK, the Forum for Responsible Research Metrics provides advocacy and leadership on the responsible use of research metrics. The Forum also supported an experiment around metrics for OA monographs as part of the Jisc open metrics lab. In HSS disciplines there is a whole range of issues with the use of citation metrics including e.g. the coverage of these disciplines within conventional databases that are commonly used for bibliometric analysis, diverse citation cultures and the importance of local language etc. However, researchers in these disciplines do cross-reference bibliographies of books to find out which works are cited, for example to understand new disciplinary spaces. At the moment, researchers would need to go to a library to do this. As open access books and the policies and models that support them are growing there are now opportunities to help researchers to do this in a quicker way. We are working with Birkbeck, University of London to develop a tool that extracts references from OA monographs and shows which items are cited in common among the selected titles. We have shown that it is feasible to build this type of tool and the next step will now be to consider the results in a community meeting in October. The project report will be published together with the recommendations from the meeting.
The recent launch of the Research on Research Institute gives all stakeholders in the sector a huge opportunity to improve our understanding of the complexities in the research and innovation system. This is particularly important as both Science Europe and the European Universities Association survey their members about research assessment practices, reflecting a widespread view that those practices may not contribute as much as they could to a healthy research culture and to research integrity. Jisc looks forward to playing our part in this exciting movement. For more information about our work in this area, please contact neil.jacobs@jisc.ac.uk or verena.weigert@jisc.ac.uk
