Developing Specific Reporting Standards in Artificial Intelligence Centred Research

Abstract

There are several emerging AI technologies that aim to enhance surgical care pathways over the coming decade. In particular, these are related to (1) diagnostics, (2) pre-operative planning, (3) intra-operative guidance and (4) surgical robotics.1 This trend has been mirrored bn the sharp increase in the number of surgical studies evaluating the use of AI. Despite this fervour, very few AI devices have reached the point of clinical implementation within surgical environments.2 This disconnect between ‘in silico bench’ and ‘bedside’ is a multifaceted issue related to technological, regulatory, and economic factors. However, this divide is also exacerbated by the variable quality of study reporting in this field; an issue perpetuated by the absence of AI-specific reporting guidelines for both pre-clinical and clinical AI studies. Poor reporting has been shown to hinder the clinical translation of otherwise promising research findings.3 One of the principle means of mitigating the risk of poor reporting is through adherence to consensus reporting standards. These tools, many of which are endorsed across biomedical journals, describe the critical information expected in research manuscripts. They typically consist of a checklist of minimally essential items, a flow diagram, and are accompanied with a longer elaboration and explanation document consisting of rationale and examples of good reporting. The enhancing the quality and transparency of health research network4 hosts a comprehensive library of reporting guidelines, with prominent examples including standard protocol items: recommendations for interventional trials (SPIRIT), consolidated standards of reporting trials (CONSORT), standards for reporting diagnostic accuracy (STARD), and Transparent Reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD). A caveat to the use of such instruments is that they are necessarily study design specific. When a study design specific tool is not available, extensions to existing tools are actively encouraged. In the absence of AI specific tools, researchers and readers have been left with the after reporting pitfalls (Supplemental Digital Content Table 1, https://links.lww.com/SLA/D514). If left unchecked, poorly reported studies can have a direct and deleterious impact upon how surgical departments shape their service. As a practical example, when reporting upon AI devices that can diagnose cancer from screening mammograms, studies should look to specify upon how the device demonstrates equitable performance across population groups, whether the system fits within existing clinical pathways, against what reference standard performance was measured against and, if it is an adaptive device, how performance alters with dataset shifts. Failing to mention this can lead to the adoption of expensive and impractical technologies that can potentially exacerbate health inequalities through the skewed diagnosis of target pathologies. Therefore, to fill this reporting guideline void, AI extensions are being undertaken for the following: 1) SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) 2013 2) CONSORT (Consolidated Standards of Reporting Trials) 2010 3) STARD (Standards for Reporting of Diagnostic Accuracy Studies) 2015 4) TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) 2015 The first in these series of reporting guidelines aim to improve the reporting of clinical trial protocols (SPIRIT-AI5) and as well as the eventual findings of the trials themselves (CONSORT-AI6). In combination, these documents are designed to enable readers to (1) understand the background, rationale, population, methods, statistical analyses, and ethico-legal considerations, (2) assist replication of key aspects of the trial (including implementation of the intervention) and (3) assist in the appraisal the study's scientific rigour. To achieve these goals