Unmet clinical need (UCN) defined

Classification of the model: diagnostic, theragnostic, predictive, prognostic, follow-up

Input from Clinicians for interpretable pipeline development. Discussion regarding choosing appropriate explainability method

Image protocol quality to be documented following the TRIAC level (Transparent Reporting of Medical Image Acquisition for a future proof radiomics). TRIAC guidelines describe five different levels of evidence for reporting imaging protocols. Level 0 indicates that the protocol has not been formally approved with a reference number; Level 1 indicates that the protocol has been approved with a reference number in the archive of the department; Level 2 indicates that the protocol has been approved with formal quality assurance (recommended minimum level for prospective trials); Level 3 indicates that the protocol is established internationally and has been published in guideline documents and peer-reviewed papers; Level 4 indicates that the protocol is Future proof i.e., the protocol follows TRIAC Level 3, FAIR principles and retains raw data.

Hardware’s used described, image reconstruction method specified

Preprocessing of the images

Imaging at multiple time points - collect individuals’ images at additional time points. Analyze feature robustness to temporal variabilities (e.g., organ movement, organ expansion/shrinkage)

Inclusion and exclusion criteria or boundaries of the model defined (e.g. a CT with 6 mm slice thickness cannot be analyzed)

The diversity and distribution across diverse patient groups in the datasets should be reported at training and testing to identify potential biases and apply appropriate corrective measures

Use of post-processing harmonization to reduce multi-center acquisition variability e.g. Combat for HCR and CycleGANs for DL

Method and statistical plan pre-registered on a public platform (e.g. www.osf.io)

Training dataset coming from one center, two centers, three centers, or more

Multiple segmentations - possible actions are segmentation by different physicians/algorithms/software, perturbing segmentations by (random) noise, segmentation at different breathing cycles. Analyze feature robustness to segmentation variabilities

Cut-off analyses - determine risk groups by either median, a previously published cut-off, or report a continuous risk variable or published method. Reduces the risk of reporting overly optimistic results

Random permutations to assess the risk of overfitting. Randomize the input variable to get ideally an AUC not different than 0.5 and therefore assess the risk of overfitting.

Investigate both handcrafted radiomics and deep learning, or a combination thereof, in an ensemble. Radiomics features may also help in the interpretability of deep learning

Quality Management System

Discrimination statistics - report discrimination statistics (e.g., C-statistic, ROC curve, AUC) and their statistical significance (e.g., p-values, confidence intervals). One can also apply a resampling method (for example, bootstrapping, cross-validation).

Calibration statistics - report calibration statistics (e.g., Calibration-in-the-large/slope, calibration plots) and their statistical significance (e.g., p-values, confidence intervals). One can also apply a resampling method (for example, bootstrapping, cross-validation).

Comparison with previously published radiomics signatures and models

Validation - the validation is performed without retraining and adaptation of the cut-off value, providing crucial information about credible clinical performance.

Prospective study registered in a trial database (real-world or In Silico), with sample size calculation - provides the highest level of evidence supporting the clinical validity and usefulness of the radiomics biomarker

Algorithm tested in a clinical environment e.g. a department of Radiology or Nuclear medicine

Evaluation should also be carried out concerning the sources of discriminative biases that have been identified

Detect and discuss biological correlates - demonstration of phenotypic differences (possibly associated with underlying gene-protein expression patterns) deepens understanding of radiomics and biology

Details on the intrinsic or post-hoc interpretability method or uncertainty estimation method utilized (e.g. attribution maps, SHAP analysis). Evaluation of the explanations using in-silico trials or clinicians.

Comparison to 'gold standard' - assess the extent to which the model agrees with/is superior to the current ‘gold standard’ method (e.g. Dr. evaluation, TNM-staging for survival prediction, Dr. Assessment). This comparison shows the added value of radiomics

Potential clinical utility - report on the current and potential application of the model in a clinical setting (e.g., decision curve analysis)

Cost-effectiveness analysis - report on the cost-effectiveness of the clinical application (e.g., QALYs generated).

Level of automation for the clinical practice.

At level 0 (No Automation), a clinician performs the clinical task without using the radiomics model.

At level 1 (Clinical Assistance), the clinician uses the radiomics model’s prediction for a part of the clinical task.

At level 2 (Partial Automation), the clinician considers the radiomics model’s prediction for the clinical task before making the final recommendation.

At level 3 (Conditional Automation), the radiomics model provides the predictions for the clinical task under supervision and the clinician can intervene at any time.

At level 4 (High Automation), the radiomics model provides the predictions and the clinician’s intervention is required for special (out-of-distribution) cases.

At level 5 (Full Automation), the radiomics model provides predictions for the clinical task without human intervention.

The algorithm, source code, and coefficients are made publicly available. Add a table detailing the different versions of software & packages used.

Details on the intrinsic or post-hoc interpretability method or uncertainty estimation method utilized (e.g. attribution maps, SHAP analysis). Evaluation of the explanations using in-silico trials or clinicians.

Define strategy to update models (frequency, approach, access to data etc)