Software as a Medical Device now constitutes a substantial share of new medical device approvals, and the algorithms within these products span the full range from classical regression to deep learning. In the UK, software qualifying as a medical device falls under the Medical Devices Regulations 2002 as amended, with the MHRA providing specific guidance on software and artificial intelligence as a medical device. In the EU, standalone software meeting the medical device definition is regulated under the MDR, with IVD software falling under the IVDR. The regulatory pathway is determined by intended purpose and risk classification, not by the modelling technique employed.
Defining Traditional Statistical Algorithms and Machine Learning in Medical Device Software
The relevant distinction is between algorithms whose functional form is fixed by the manufacturer and algorithms that learn their form from data. A third category, mechanistic algorithms, sits alongside both and is worth naming because it also appears inside medical devices.
Traditional statistical algorithms impose structure. Logistic regression specifies that the log-odds of the output are a linear combination of the inputs. Cox proportional hazards models specify that the hazard ratio between any two patients remains constant over time. The manufacturer defines this structure in advance, on the basis of established statistical or clinical knowledge, and the algorithm estimates a small number of parameters. QRISK and CHA₂DS₂-VASc, both widely deployed in clinical practice, are built this way. The defining characteristic for regulatory purposes is traceability: each parameter has a defined interpretation, and the algorithm’s assumptions can be stated explicitly and verified during analytical validation.
AI or machine learning algorithms do not impose structure. A neural network is a composition of matrix operations and nonlinear transformations whose weights are fitted by gradient descent. A random forest is an ensemble of decision trees grown to maximise predictive accuracy. The algorithm discovers the input-output relationship from the training data. The manufacturer controls only indirect parameters such as network architecture, learning rate or number of trees, collectively the hyperparameters. The defining characteristic for regulatory purposes is that the functional form is not directly inspectable, which has consequences for verification, traceability and change control.
The boundary between the two categories is less sharp than commonly assumed. Penalised regression such as the lasso shrinks coefficients towards zero and is therefore a form of regularised flexibility. Gradient-boosted ensembles with shallow trees may be less flexible than a heavily parameterised polynomial regression. The axis that matters for risk assessment is model capacity, not the label on the box.
Mechanistic algorithms occupy a third position. These are deterministic transformations built from physical or physiological theory rather than fitted to population data: CT image reconstruction applies the Radon transform, and target-controlled infusion pumps solve pharmacokinetic differential equations in real time. They are validated as engineering systems rather than through clinical evaluation of inference, and their verification pathway differs accordingly.
For the remainder of this post, “traditional” refers to statistical algorithms with explicit functional forms, and “AI” refers to flexible learners, both in the context of software embedded in or operating as a regulated medical device.
Where AI and Traditional Algorithms Differ: Overfitting, Calibration and Deployment Risks
Overfitting risk and performance claims in machine learning devices
A logistic regression with eight inputs has limited capacity to fit noise in the development dataset. A neural network with millions of parameters can model idiosyncrasies of that dataset with great precision. This is a difference of degree, but the degree is large enough to matter for regulatory purposes.
For traditional algorithms, internal validation using bootstrapping or cross-validation is good practice and sufficient to support performance claims in many cases. For high-capacity models, such validation is essential. The gap between apparent performance on training data and the optimism-corrected estimate is typically much wider, and the corrected estimate is the only defensible figure for a performance claim in technical documentation.
Calibration drift and clinical benefit claims
Flexible models frequently produce poorly calibrated outputs without deliberate intervention. Van Calster and colleagues described calibration as the Achilles heel of predictive analytics, and models developed on modest samples are the worst affected.
The mechanism is structural. A model with spare capacity fits noise alongside signal, and fitted noise pushes outputs towards the extremes: too close to 0 or 1. This appears as a calibration slope below 1 when the model is evaluated against independent data. Regularisation mitigates the problem. So does careful cross-validation for hyperparameter tuning, and explicit calibration assessment during clinical evaluation. None of this happens automatically, and the manufacturer must demonstrate that it has been done.
The regulatory consequence is direct. Miscalibration undermines the clinical benefit claim. If the intended purpose states that the device supports decisions at a specific risk threshold, and the algorithm systematically over- or under-estimates risk, the clinical benefit is not delivered. This connects calibration to the general safety and performance requirements of MDR Annex I, under which a device shall achieve the performance intended by its manufacturer.
Interpretability, traceability and technical documentation
Regression coefficients can be read directly and documented. The coefficient on systolic blood pressure in a cardiovascular risk algorithm has an immediate, specifiable interpretation that can be written into the technical file and reviewed by a notified body or approved body.
Outputs from a deep neural network generally require post-hoc explanation methods such as SHAP. These tools identify which inputs the model relied on for a given prediction, but they do not establish whether that reliance is clinically sound, and different explanation methods can produce different accounts of the same model. From a quality management perspective, this creates a documentation burden: the manufacturer must explain not only what the algorithm does but how that explanation was generated and why it is trustworthy.
Unstructured input data and pipeline complexity
Traditional algorithms consume structured variables: patient demographics, laboratory values, physiological measurements. AI systems frequently consume raw images, free text, waveforms or genomic sequences. This extends the pipeline that must be verified and validated. Preprocessing, augmentation and acquisition protocols all become part of the algorithm, and each introduces potential failure modes.
Scanner drift between hospitals, site-specific imaging artefacts, differences in acquisition protocol between deployments: none of this has an analogue in a regression on structured clinical variables. For a manufacturer, this means the intended use environment must be characterised more precisely, and the algorithm’s sensitivity to these variations must be assessed during performance evaluation.
Sample size and clinical evidence generation
The rule of thumb of ten events per variable is inadequate even for regression models, as van Smeden and colleagues demonstrated. Required sample size depends on the number of candidate inputs, the outcome proportion and the expected strength of input-output associations.
For flexible models the requirements are higher. More parameters demand more data before performance stabilises, and this has direct consequences for the clinical evidence plan. The manufacturer must demonstrate that the development dataset was sufficient to support the performance claims made in the technical documentation, and for AI systems this demonstration is more demanding than for a regression model with the same intended purpose.
Deployment failure modes: data leakage, shortcut learning and dataset shift
Three failure modes are specific to or substantially worse for flexible models deployed in regulated products.
Data leakage occurs during development when information from the test set contaminates the training process. The most common form is preprocessing leakage: scaling or imputation applied to the full dataset before the train-test split. Performance is inflated, and the test set is no longer truly independent. It is a pipeline error rather than a modelling error, and it is common enough that any surprisingly strong performance claim warrants a leakage audit before it enters technical documentation.
Shortcut learning occurs when the model relies on a spurious feature rather than the genuine clinical signal. Documented cases in medical imaging include models that learned to detect hospital-specific text labels or the presence of a ruler in the image rather than the pathology itself. Performance on the test set appears excellent, because the spurious feature is present there too. Performance degrades when the product encounters data from a different source. For a manufacturer, this is a post-market surveillance risk: the product may perform as claimed in the validation setting and fail in the deployment setting.
Dataset shift refers to changes in the data distribution between development and deployment. Clinical practice changes over time. Patient populations differ between sites. Outcome prevalence differs between settings. All degrade performance, and all are insidious because the algorithm continues to produce plausible-looking outputs that are systematically wrong. This drives the post-market clinical follow-up requirements discussed below.
What Does Not Differ: Clinical Evaluation and Regulatory Requirements for All SaMD
The regulatory framework itself is substantially independent of modelling technique.
The qualification question is the same. Does the software have an intended medical purpose that brings it within the definition of a medical device? If yes, the regulatory pathway follows from the intended purpose and risk classification, not from the algorithm’s architecture. Under EU MDR, software that provides information used to inform diagnostic or therapeutic decisions is classified as Class IIa as a minimum under Rule 11, escalating according to the consequences of an incorrect output. Under the IVDR, software that provides information derived from in vitro diagnostic data falls within the IVD framework, with companion diagnostics typically classified as Class C. In the UK, the equivalent analysis is conducted under the UK MDR 2002 with MHRA guidance on software and AI as a medical device.
The clinical evaluation requirements are the same. MDR Annex XIV Part A requires a clinical evaluation for all devices, regardless of classification or algorithm type. The evaluation must demonstrate conformity with relevant general safety and performance requirements, based on clinical data. The IMDRF framework for SaMD clinical evaluation, which provides internationally harmonised guidance used by regulators including the MHRA and EU notified bodies, sets out three components: valid clinical association, analytical validation and clinical validation. A regression model and a neural network are assessed against the same three components.
The performance metrics are the same. Discrimination is quantified by AUC or C-index. Calibration is assessed through calibration curves, slopes and intercepts. Clinical utility is evaluated where relevant. Decision curve analysis applies to any model whose output informs a clinical decision. None of these metrics distinguishes between predictions generated by maximum likelihood estimation and those generated by gradient descent.
The evidence standard for market access is the same. Independent evaluation of performance, whether through a clinical investigation or through rigorous retrospective validation against an appropriate reference standard, is expected for any algorithm making a performance claim. Differences in case-mix, measurement protocols and outcome ascertainment between development and deployment settings degrade performance for all algorithm types without exception.
The clearest confirmation that the framework is shared comes from the reporting guidelines. TRIPOD+AI was written as a single guideline covering regression and machine learning methods alike, because the underlying questions asked by reviewers and regulators are identical. Only the methodological detail required at each step differs.
UK and EU Regulatory Frameworks: MDR Rule 11, IVDR and IMDRF SaMD Guidance
For UK and EU manufacturers, several instruments bear directly on algorithm-containing software.
The UK Medical Devices Regulations 2002, as amended, provide the current framework for Great Britain, with CE marking recognised for a transitional period. The MHRA’s guidance on software and artificial intelligence as a medical device addresses qualification, classification and lifecycle requirements for algorithm-containing products.
The EU Medical Devices Regulation 2017/745 governs software meeting the medical device definition in the EU market. Rule 11 of Annex VIII provides the classification framework: software providing information used for diagnostic or therapeutic decisions is Class IIa, rising to Class IIb where such decisions may cause serious deterioration of a person’s state of health or a surgical intervention, and Class III where death or irreversible deterioration of a person’s state of health may result from an incorrect output. Software intended to monitor physiological parameters is Class IIa, or Class IIb for vital parameters where the nature of variations is such that they could result in immediate danger to the patient.
The EU In Vitro Diagnostic Regulation 2017/746 covers software that processes in vitro diagnostic data. Companion diagnostics, which typically incorporate algorithmic analysis of molecular or genomic data, are classified as Class C under Rule 3(f).
The IMDRF framework for SaMD clinical evaluation provides the internationally harmonised structure for generating clinical evidence, built around valid clinical association, analytical validation and clinical validation. This framework applies regardless of whether the algorithm is a regression or a neural network.
Post-market requirements apply to all algorithm-containing devices. Post-market surveillance under MDR Article 83, post-market clinical follow-up under Annex XIV Part B, and periodic safety update reports where applicable: these mechanisms exist to detect the dataset shift and performance degradation described above. For AI systems that are updated or retrained after deployment, change control becomes a central regulatory concern, since modifications affecting safety or performance may require new conformity assessment.
Choosing Between Machine Learning and Statistical Algorithms for a Medical Device
When is a traditional algorithm sufficient for a regulated product? When the input data are structured, the development dataset is modest, and the input-output relationships are plausibly approximated by a parametric model on the appropriate scale. This description covers a large fraction of clinical algorithms currently embedded in medical software, and the traceability advantages simplify both verification and documentation.
When is a flexible learner justified? When the input is unstructured, when the available dataset is large, when the true relationships are likely nonlinear and interactive in ways that cannot be specified in advance, or when there is a specific performance requirement that a parametric model demonstrably cannot meet. Even then, the comparison should be against a well-built traditional baseline rather than a weak one. The Christodoulou review found no evidence of superior machine learning performance over logistic regression for clinical prediction, and noted that many of the comparison studies suffered from poor methodology and reporting, particularly in their validation procedures. A fair, competently built comparison strengthens the justification for the more complex approach.
The regulatory consequence of algorithm choice is primarily a documentation and verification burden rather than a fundamental difference in pathway. A manufacturer choosing a neural network over a regression model for the same intended purpose accepts additional work in calibration assessment, interpretability documentation, and more frequent change control events arising from model updating and retraining. The justification for accepting that burden should be articulated in the clinical evaluation plan and risk management file.
Algorithm Choice, Evidence Requirements and Lifecycle Burden
The intended purpose of the device determines what evidence is required. Algorithm flexibility determines how much work is needed to generate that evidence.
A regression model and a neural network embedded in medical devices with the same intended purpose are assessed against the same general safety and performance requirements, held to the same standards of clinical evidence, and subject to the same post-market obligations. The neural network introduces additional challenges: overfitting, shortcut learning, dataset shift, data leakage, poor calibration without intervention, and a documentation burden arising from reduced traceability. These are manageable, but managing them requires deliberate effort throughout the product lifecycle, from design input through post-market surveillance.
The distinction between traditional and AI algorithms in medical devices is therefore real but bounded. It is a difference in the rigour required at each stage of evidence generation and lifecycle management, not a difference in the regulatory framework itself. Manufacturers selecting between the two should weigh the performance benefit of flexibility against the documentation, verification and post-market burden that flexibility carries, and should record that reasoning in the design history file.
References
- Christodoulou E, et al. A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. Journal of Clinical Epidemiology. 2019;110:12–22.
- Collins GS, Moons KGM, Dhiman P, et al. TRIPOD+AI statement. BMJ. 2024;385:e078378.
- European Commission. Regulation (EU) 2017/745 on medical devices. Annex I; Annex VIII, Rule 11; Annex XIV; Article 83.
- European Commission. Regulation (EU) 2017/746 on in vitro diagnostic medical devices. Rule 3.
- IMDRF. Software as a Medical Device (SaMD): Clinical Evaluation. 2017. IMDRF/SaMD WG/N41FINAL:2017.
- Medicines and Healthcare products Regulatory Agency. Software and artificial intelligence (AI) as a medical device. GOV.UK, 2025.
- van Smeden M, Moons KGM, de Groot JAH, et al. Sample size for binary logistic prediction models: beyond events per variable criteria. Statistical Methods in Medical Research. 2019;28(12):2455–2474.
- Van Calster B, McLernon DJ, van Smeden M, Wynants L, Steyerberg EW. Calibration: the Achilles heel of predictive analytics. BMC Medicine. 2019;17:230.
