AI Algorithms now sit at every point in the patient pathway. They identify who has a disease, forecast what will happen next, and track change over time. The statistical machinery needed to validate each of these three types is fundamentally different. A model that classifies a patient as “disease-positive” at a single timepoint is not validated the same way as one that predicts 5-year survival. Neither maps neatly onto the tools needed to judge whether a biomarker is genuinely tracking disease activity longitudinally. This post unpacks the three categories, explains where their validation requirements diverge, and flags the methodological pitfalls that reviewers and regulators see most often.
The conceptual framework used here draws on the FDA–NIH BEST (Biomarkers, EndpointS, and other Tools) glossary. This glossary formally distinguishes seven biomarker categories, including diagnostic, prognostic, and monitoring, precisely because their evidentiary requirements differ (FDA-NIH Biomarker Working Group, 2016).
Diagnostic Algorithms: Answering “Does This Patient Have the Disease?”
A diagnostic algorithm detects or confirms the presence of a disease or condition of interest, or identifies individuals with a particular disease subtype (FDA-NIH Biomarker Working Group, 2016). The defining feature is that the algorithm operates at a single point in time. Its output is categorical: disease present or absent, subtype A or subtype B. This output is then compared against a reference standard.
Classic examples include imaging-based classifiers that flag malignant versus benign lesions, symptom-checking tools that triage patients into “refer” or “reassure” pathways, and laboratory panels that identify a specific infectious agent. What unites them is the cross-sectional nature of the question. At the moment of testing, does the target condition exist?
Prognostic Algorithms: Answering “What Is Likely to Happen to This Patient?”
A prognostic algorithm identifies the likelihood of a future clinical event, disease recurrence, or progression in patients who typically already have the condition of interest (FDA-NIH Biomarker Working Group, 2016). Unlike diagnostic tools, prognostic models operate on a temporal horizon. Six-month mortality. Ten-year cardiovascular risk. Probability of relapse. The output is a probability or risk score rather than a binary label.
The output is inherently a forecast. This matters for validation because the ground truth is not available at the time the prediction is made. It only emerges as follow-up accrues. Prognostic models therefore require longitudinal cohort data with well-defined outcome definitions and time-to-event methodology when outcomes occur over variable follow-up.
Disease Monitoring Algorithms: Answering “Is the Disease Changing Over Time?”
A monitoring algorithm tracks the status of a disease or medical condition over time, including the patient’s response to treatment or exposure to a medical substance or environmental agent (FDA-NIH Biomarker Working Group, 2016). Monitoring is fundamentally longitudinal and within-subject. The relevant signal is not the absolute value at any one visit. It is the change relative to that individual’s own baseline and the natural variability of the measurement.
Examples include tumour response assessment using RECIST criteria, where changes in the sum of longest diameters of target lesions determine response categories: complete response, partial response, stable disease, progressive disease (Ko et al., 2021). Viral load monitoring in HIV. HbA1c tracking in diabetes. Liquid-biopsy platforms that sample blood at multiple timepoints to detect molecular progression. The analytical question shifts from “is the patient positive?” to “has this patient changed more than measurement noise would allow?”
Side-by-Side Comparison
| Dimension | Diagnostic | Prognostic | Monitoring |
|---|---|---|---|
| Clinical question | Does the patient have the disease now? | What is the risk of a future outcome? | Is the disease changing over time? |
| Timepoint structure | Single cross-sectional assessment | Baseline prediction, future outcome | Serial measurements within-subject |
| Typical output | Binary class / subtype label | Probability or risk score over a defined horizon | Change score, rate of change, response category |
| Reference standard | Concurrent gold-standard test / biopsy / expert adjudication | Longitudinally observed clinical outcome | Clinically defined change (e.g. progression, response) |
| Core performance metrics | Sensitivity, specificity, PPV, NPV, ROC-AUC | Harrell’s C-index / time-dependent AUC | Concordance of change with clinical change; limits of change |
| Calibration focus | Threshold alignment with prevalence | Predicted vs observed risk across the full risk range | Within-subject biological vs analytical variability |
| Key reporting guideline | STARD / STARD-AI / QUADAS | TRIPOD / TRIPOD+AI | Context-specific (e.g. RECIST) |
| Primary regulatory framing | Analytical + clinical validity vs reference standard | Clinical validity + clinical utility (DCA) | Analytical validity of change + clinical validity of response definition |
A Note on Traditional Versus AI Algorithms
Everything above applies whether the underlying model is a logistic regression or a deep neural network. But the terms deserve a brief definition, because “traditional” and “AI” are often used loosely.
A traditional algorithm in this context means a statistical model with an explicit functional form chosen by the analyst. Logistic regression assumes a specific relationship between predictors and the log-odds of the outcome. A Cox proportional hazards model assumes proportional hazards over time. The analyst specifies the structure, the software estimates the coefficients, and the result is interpretable: each coefficient has a meaning, and the model’s assumptions can be checked against the data.
An AI algorithm, in the machine learning sense, is a flexible learner. Neural networks, random forests, and gradient-boosting machines do not impose a functional form. They discover one from the data, fitting millions or billions of parameters to minimise prediction error. The resulting model may perform well, but its internal reasoning is not directly readable from its parameters.
The distinction matters for validation workload, not validation framework. A diagnostic model built as a convolutional neural network is still compared against a reference standard, still reports sensitivity and specificity, and still reports per STARD-AI. A prognostic gradient-boosting model still needs its discrimination and calibration assessed per TRIPOD+AI. That guideline deliberately covers regression and machine learning methods alike, because the underlying validation questions are shared. What changes is how carefully each step must be executed: flexible models overfit more readily, need more data, and introduce failure modes such as shortcut learning that have no analogue in a regression setting.
How algorithm type changes the validation workload, in detail, is the subject of the companion post.
Validating Each Algorithm Type
Diagnostic Validation: Accuracy Against a Reference Standard
Diagnostic accuracy studies compare the index test or algorithm against a clinical reference standard in a consecutive or otherwise representative patient series. The core output is the 2×2 table. From it, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) are derived. These are typically presented alongside the receiver operating characteristic (ROC) curve and its area under the curve (AUC).
Several methodological points are non-negotiable.
The reference standard must be independent of the index test. Incorporation bias occurs when the index test result forms part of the reference standard. It artificially elevates both sensitivity and specificity.
Verification bias must be addressed. If only test-positive patients receive the reference standard, sensitivity is overstated and specificity understated.
PPV and NPV are prevalence-dependent. A test with identical sensitivity and specificity will have very different predictive values in a screening population versus a referral population. The study sample must therefore reflect the intended-use population.
Reporting must follow STARD. The STARD 2015 statement provides a 30-item checklist covering participant flow, blinding, timing between index test and reference standard, handling of missing or indeterminate results, and confidence intervals for every accuracy metric (Bossuyt et al., 2016). For AI-based diagnostic tools, the STARD-AI extension applies. It was published in Nature Medicine in 2025 following a Delphi consensus involving over 240 international stakeholders (Sounderajah et al., 2025). For systematic reviews of diagnostic accuracy, QUADAS-2 is the established risk-of-bias tool (Whiting et al., 2011). Where a review compares the accuracy of two or more tests, the QUADAS-C extension can additionally assess risk of bias in the estimated difference in accuracy between tests.
One critical insight is often missed. STARD explicitly notes that diagnostic accuracy studies never determine a test’s “true” sensitivity and specificity. At best, they provide estimates conditioned on the study population and reference standard used (Bossuyt et al., 2016). External validation in new settings is therefore essential before generalising results.
Prognostic Validation: Discrimination, Calibration, and Clinical Utility
Prognostic model validation rests on two statistical pillars that answer distinct questions. A third addresses decision-making.
Discrimination asks: does the model rank-order patients correctly? Do those who experience the outcome receive higher predicted risk than those who do not? For binary outcomes, this is captured by the AUC. For time-to-event outcomes, Harrell’s C-index is the standard measure (Ramspek et al., 2021).
Calibration asks: do predicted probabilities match observed event rates? A model can discriminate well while systematically over-predicting or under-predicting risk. Calibration is assessed via several metrics.
The calibration intercept (calibration-in-the-large) compares the average predicted risk with the observed event rate. A value of 0 indicates perfect agreement.
The calibration slope is obtained by fitting a new model in the validation dataset with the prognostic index as sole predictor. A slope of 1 indicates perfect calibration. A slope below 1 signals overfitting in the original model (Ramspek et al., 2021).
The observed-to-expected (O/E) ratio provides a measure of aggregate calibration.
The Brier score assesses overall predictive performance. Its interpretation is subtler than commonly assumed. A “fence-sitter” that predicts 0.5 for every patient scores 0.25, so 0.25 is the natural non-informative benchmark. Scores must therefore be read against the outcome’s base rate, since always predicting the base rate p yields p(1−p), which equals 0.25 only when events are equally common. A score of 0.20 may be excellent for a rare outcome and unimpressive for a balanced one. Crucially, a low Brier score does not guarantee good calibration, because the score conflates discrimination and calibration into a single number (Hoessly et al., 2026).
Calibration is generally regarded as more important than discrimination when predicted risks will inform individual clinical decisions. A well-calibrated model ensures patients are told the truth about their risk.
Clinical utility is most often quantified using decision curve analysis (DCA). This estimates the net benefit of using the model to guide decisions across a range of threshold probabilities, and compares it against treating all or treating none (Zhang, 2018). Traditional metrics like AUC measure statistical performance but ignore the decision context. DCA integrates the harm–benefit trade-off of the actions triggered by the prediction.
Reporting and design requirements follow TRIPOD, the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis statement. TRIPOD was updated as TRIPOD+AI in 2024. This is a 27-item checklist covering regression and machine-learning approaches alike, and it explicitly covers prediction models for diagnostic, prognostic, monitoring and screening purposes regardless of modelling approach (Collins et al., 2024).
Internal versus external validation must be distinguished.
Internal validation, using bootstrapping or repeated k-fold cross-validation, estimates optimism-corrected performance within the development dataset. It is mandatory when reporting any new model. Simple split-sample validation is now discouraged as inefficient and unstable, particularly in smaller datasets. Steyerberg’s comparison found it gave overly pessimistic performance estimates with large variability. Bootstrapping is the recommended alternative (Steyerberg et al., 2001).
External validation evaluates performance in data not used for development, ideally from a different institution, time period, or geographical region.
Sample size deserves explicit planning rather than reliance on rules of thumb. Early guidance suggested that external validation of a prognostic model requires a minimum of 100 events, preferably 200 or more, with a related rule of thumb of at least 100 events and 100 non-events (Collins et al., 2016). However, simulation work has shown that such blanket guidance is imprecise. The precision of performance estimates depends on the distribution of the model’s linear predictor and the case-mix of the validation population, not just event counts (Snell et al., 2021). Simulation-based sample size calculations are now preferred, and they frequently demand more data than the old rules imply. This is particularly true for calibration. In one worked example, estimating the calibration slope with adequate precision required 949 participants (408 events), far more than the 347 participants (149 events) needed for the c-statistic (Riley et al., 2024). When planning a validation study, calibration, not discrimination, is usually the binding constraint.
Missing data must be handled transparently, with the method matched to the missingness pattern. See Pitfalls below.
Monitoring Validation: Change, Variability, and Limits
Monitoring algorithms are the least standardised of the three to validate. Part of the reason is that the question, “has this patient truly changed?”, is statistically distinct from classification and risk prediction.
The validation architecture has three layers.
Analytical validation of the measurement itself. Before a biomarker can be used to monitor change, its analytical performance must be characterised. This includes precision, repeatability, and reproducibility across laboratories and batches.
Within-subject biological variability and reference change values. Because monitoring is inherently within-subject, the key statistical quantity is the reference change value (RCV). This is the change between two measurements that exceeds what would be expected from combined analytical and within-subject biological variation alone. For a biomarker with analytical coefficient of variation CVₐ and within-subject biological coefficient of variation CVᵢ, the RCV at a two-sided 95% level is:
RCV=1.96×2×CVa2+CVi2 Deriving this requires repeated-measurement data in stable patients to estimate the within-subject standard deviation (Clinical Chemistry, 2025). Without it, clinicians cannot distinguish true disease progression from measurement noise.
Clinical validity of the change definition. The algorithm’s definition of “meaningful change” must be validated against a clinical anchor. Does a biomarker change of the defined magnitude correspond to clinically recognised progression, response, or relapse? In oncology, RECIST 1.1 provides a standardised, validated framework. Changes in the sum of longest diameters of target lesions, assessed at predefined timepoints, determine response categories: complete response (CR), partial response (PR), stable disease (SD), and progressive disease (PD). RECIST was explicitly developed and validated to provide objective, reproducible assessment of treatment effect, and it remains the standard framework for response assessment in solid tumour trials (Ko et al., 2021). In other therapeutic areas, longitudinal data must be collected and analysed, often using mixed-effects models or joint models of longitudinal and survival data, to link biomarker trajectories to clinical outcomes.
Longitudinal data requirements are substantial. Monitoring validation requires serial samples in the same individuals over clinically meaningful intervals. There must be enough stable patients to characterise variability and enough progressing or responding patients to anchor the change thresholds. This makes monitoring studies logistically heavier than single-timepoint diagnostic studies. Cross-sectional data can establish population reference ranges, but they cannot characterise within-subject variability, which is the critical quantity for judging whether an observed change is real.
Common Pitfalls in Validation Practice
Several recurring errors undermine the validity of published algorithms across all three categories.
Using accuracy metrics appropriate to one algorithm type for another. Reporting AUC for a monitoring biomarker, or the C-index for a diagnostic classifier, signals a category confusion. AUC quantifies the ability to rank-order individuals at a single timepoint. Monitoring algorithms instead ask whether an individual has changed relative to their own baseline. This requires change-based metrics: concordance of change scores, reference change values, limits of change. Each metric family presupposes a specific data structure. Applying the wrong family answers a different question from the one the algorithm was built for.
Neglecting calibration in prognostic models. Discrimination alone is insufficient. A model can rank patients perfectly while telling every patient their risk is 30% too high. Calibration assessment, including slope, intercept, and O/E ratio, is mandatory. The widely used Hosmer–Lemeshow test is now discouraged as an overall measure because it depends heavily on sample size (Ramspek et al., 2021).
Equating internal with external validation. Bootstrapped or cross-validated performance is an internal check. It does not demonstrate transportability. Differences in case-mix, measurement protocols, and outcome ascertainment between development and deployment settings degrade real-world performance in ways internal validation cannot detect (Snell et al., 2021).
Ignoring prevalence when interpreting predictive values. PPV and NPV vary with disease prevalence. Accuracy estimates from a tertiary-referral population do not transfer to primary care without recalibration of expectations.
Treating all missing data alike. The appropriate method depends on where and why data are missing. When missingness is confined to the outcome, complete-case analysis and multiple imputation are equivalent, and complete-case analysis is preferred for its simplicity and robustness. When predictors are incomplete under missing-at-random assumptions, multiple imputation is generally preferred. Complete-case analysis discards information and can bias estimates when missingness is related to patient characteristics or the outcome itself (Hughes et al., 2019). Contrary to widespread belief, complete-case analysis is not uniformly inferior.
Underpowered external validation. Validation studies with too few events, below the 100-event floor and often well above it when precise calibration estimates are required, produce C-index and calibration estimates with confidence intervals too wide to support firm conclusions about model adequacy (Collins et al., 2016; Riley et al., 2024).
Confusing prognostic with predictive biomarkers. A prognostic biomarker informs outcome regardless of therapy. A predictive biomarker indicates likely benefit from a specific treatment relative to baseline (FDA-NIH Biomarker Working Group, 2016). Mislabelling one as the other leads to inappropriate clinical and regulatory decisions.
Summary
Diagnostic, prognostic, and monitoring algorithms occupy distinct nodes in the clinical pathway, and their validation frameworks must respect that structural difference. Diagnostic algorithms demand rigorous cross-sectional comparison against an independent reference standard, reported per STARD. Prognostic models require longitudinal data, assessment of both discrimination and calibration, decision-curve analysis for clinical utility, and formal external validation per TRIPOD. Sample sizes should be driven by the demands of calibration estimation. Monitoring algorithms rest on analytical validation of the measurement, characterisation of within-subject variability through reference change values, and clinical anchoring of change thresholds. This is a three-layer process for which standardised frameworks like RECIST exist only in select therapeutic areas.
The unifying principle is simple. The validation design must mirror the intended clinical use. A model validated to answer one question, whether presence, future risk, or change, cannot be assumed to answer another without evidence specific to that context.
References
- Bossuyt PM, et al. STARD 2015 guidelines for reporting diagnostic accuracy studies. BMJ Open. 2016;6(11):e012799.
- Estimating reference change values using routine patient data. Clinical Chemistry. 2025;71(2):307.
- Collins GS, et al. Sample size considerations for the external validation of a multivariable prediction model. Statistics in Medicine. 2016. (PMC4738418)
- Collins GS, Moons KGM, Dhiman P, et al. TRIPOD+AI statement. 2024. tripod-statement.org
- FDA-NIH Biomarker Working Group. BEST (Biomarkers, EndpointS, and other Tools) Resource. 2016. NCBI Bookshelf NBK326791 (Glossary: NBK338448).
- Hoessly L, et al. On misconceptions about the Brier score in binary prediction models. 2026. (PMC12818272)
- Hughes RA, et al. Accounting for missing data in statistical analyses: multiple imputation and its alternatives. International Journal of Epidemiology. 2019;48(4):1294–1304.
- Ko CC, et al. Imaging biomarkers for evaluating tumour response. 2021. (PMC8252278)
- Ramspek CL, et al. External validation of prognostic models. Clinical Kidney Journal. 2021;14(1):49–58.
- Riley RD, Snell KIE, Archer L. Evaluation of clinical prediction models (part 3): calculating the sample size required for an external validation study. BMJ. 2024;384:e074821.
- Snell KIE, Archer L, Ensor J, et al. External validation of clinical prediction models: simulation-based sample size calculations were more reliable than rules-of-thumb. Journal of Clinical Epidemiology. 2021;133:79–89.
- Sounderajah V, et al. The STARD-AI reporting guideline for diagnostic accuracy studies using artificial intelligence. Nature Medicine. 2025.
- Steyerberg EW, et al. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. Journal of Clinical Epidemiology. 2001.
- Whiting PF, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Annals of Internal Medicine. 2011;155(8):529–536.
- Zhang Z. Decision curve analysis: a technical note. Annals of Translational Medicine. 2018.
