Biostatistics Blog

Series: Advanced Biostatistics for MedTech: Bridging Clinical Evaluation and EngineeringThe Regulatory Shift Toward Algorithmic ValidationThe integration of machine learning (ML) into Software as a Medical Device (SaMD), has fundamentally altered the regulatory landscape. The industry’s transition from the use of traditional statistical modeling (SM) to machine learning (ML) within device technology, has necessitated a paradigm shift in how predictive algorithms are validated. Under the EU Medical Device Regulation (MDR 2017/745) and the FDA’s evolving framework for AI/ML-based SaMD, the regulatory expectation of safety, efficacy, and generalisability remains unchanged.Harrell (2018) distinguishes between SM and ML not by dataset size or dimensionality (both approaches can operate in high-dimensional settings) but by the following three considerations:Uncertainty (whether a probability model for the data is specified),Structure (whether additivity of predictor effects is assumed), andEmpirical emphasis (whether high-order interactions are pre-specified or discovered algorithmically).One constraint he identifies is information content rather than dimensionality. Where additive effects dominate, SM typically requires on the order of twenty events per candidate predictor, whereas ML may require in the region of two hundred. Simulation evidence found that logistic regression reached a stable AUC at approximately 20-50 events per variable. In contrast, random forest, support vector machine and neural network models still exhibited instability and optimism bias beyond 200 events per variable (van der Ploeg et al., 2014).This distinction is critical for medtech. ML achieves its strongest results in domains with high signal-to-noise ratios and knowable ground truths, such as image recognition. Clinical outcome prediction involves a large irreducible component of variation. The proliferation of continuous monitoring devices (e.g., continuous glucose monitors, wearable ECGs) is routinely offered as evidence that the data-volume problem has been solved. High-frequency sampling, however, inflates the number of data rows without proportionately increasing the effective sample size. For example, a fourteen-day CGM trace yields roughly twenty thousand readings from a single patient; a single-lead ambulatory ECG patch sampling at 200Hz yields upwards of two hundred million over the same period. If the endpoint is a clinical event the effective sample size is the number of patients and events, not rows. The defensible application of ML in SaMD therefore relies on the genuine presence of strong, non-pre-specifiable interactions (such as those in waveform morphology or image-derived features) rather than data volume alone.Collinearity and Regularisation: Beyond PLS and RidgeUnder the taxonomy just described, penalised regression (such as ridge, LASSO, and the Elastic Net) falls on the statistical modelling side of the divide. The usual LASSO assumes that every predictor acts linearly and that the model is fully additive. It is a regularised regression, not an algorithmic learner. The methods in this section are best understood as high-dimensional statistical modelling, with the tree-based ensembles discussed later constituting the ML component proper. The distinction determines which sample-size heuristics apply, which structural assumptions must be verified and what a reviewer is entitled to ask about functional form.In high-dimensional medtech data, multicollinearity is the norm. Standard Ordinary Least Squares (OLS) regression fails when the number of regressors p approaches or exceeds the sample size n, as the matrix X^TX becomes singular or ill-conditioned.Pardo (2023) discusses classical remedies to this, including Partial Least Squares (PLS) and Ridge Regression. Ridge regression adds an L_2 penalty to the loss function:\min_{\beta} \left\{ \sum_{i=1}^{n} (y_i - x_i^T \beta)^2 + \lambda \sum_{j=1}^{p} \beta_j^2 \right\}While Ridge regression shrinks coefficients toward zero, it does not perform feature selection. All p regressors remain in the model. In high-dimensional SaMD applications clinical interpretability and regulatory traceability are paramount. Retaining hundreds of non-informative features is, therefore, undesirable.The Least Absolute Shrinkage and Selection Operator (LASSO) regression addresses this by imposing an L_1 penalty:\min_{\beta} \left\{ \sum_{i=1}^{n} (y_i - x_i^T \beta)^2 + \lambda \sum_{j=1}^{p} |\beta_j| \right\}The L_1 penalty forces some coefficients to exactly zero, which performs implicit feature selection. The LASSO estimate corresponds to the posterior mode under independent double-exponential (Laplace) priors on the coefficients.For SaMD the algorithm must be traceable and defensible. LASSO regression offers a mathematically defensible method for isolating a sparse set of features that are sufficient for prediction, however, this selection should not be overstated. LASSO is model-selection consistent only under restrictive conditions on the design matrix (Zhao & Yu, 2006). The selected set can be markedly unstable across bootstrap resamples when predictors are correlated. It does not inherently identify the physiologically or causally critical features of the data. Where a stable and reportable feature set is required, selection stability should be quantified. This can be achieved by refitting the model across resamples and reporting the selection frequencies, or through the stability selection framework (Meinshausen & Bühlmann, 2010).The Elastic Net regularisation regression combines L_1 and L_2 penalties. It is often preferable when features are highly correlated (e.g., adjacent time-points in a physiological waveform) as it selects groups of correlated variables, where LASSO regression might arbitrarily pick one. Both these choices have trade-offs. The objection to Ridge regression (that every regressor survives) applies in weakened form to the Elastic Net. Sparsity and stability are in tension. The appropriate operating point is a risk-based judgement of how much model transparency the intended use, risk classification, and clinical decision context actually demands.Evaluating Predictive Performance: Leakage, Metrics, and ValidationData leakage is among the more consequential and least visible defects in SaMD model development. It occurs when information from the validation or test data inadvertently influences the model training process. Generally, it produces performance estimates that are optimistic and fail to survive contact with the intended use population. The FDA’s December 2024 final guidance is pointed on this matter, setting out expectations for the purpose, timing, characteristics, and independence of test data.Standard k-fold cross-validation partitions the data into k subsets, training on k-1 and testing on the held-out fold. In SaMD development data preprocessing (e.g., feature scaling, imputation, and even the LASSO penalty parameter \lambda) is frequently performed on the entire dataset before cross-validation. This violates the principle that the model must be completely blind to the test data. To prevent this preprocessing leakage, all preprocessing and hyperparameter tuning must be nested within the cross-validation loop. In R, this is enforced using the caret or tidymodels frameworks, where preprocessing recipes are estimated solely on the training folds and then applied to the validation fold.Preprocessing leakage is not, however, the form of leakage most likely to invalidate a continuous-monitoring device submission. Two structural varieties matter more. The first is subject-level leakage. Continuous monitoring generates many observations per patient. Where those observations are allocated to folds at random, records from the same individual appear in both training and validation. Here the model is rewarded for recognising the patient rather than the pathology. The consequences of this are not marginal. Saeb et al. (2017) demonstrated that record-wise cross-validation of mobile-sensor models yields accuracies substantially above the subject-wise estimates obtained from the identical data, and it is the subject-wise estimate that approximates the intended use. Partitioning must therefore occur at the level of the independent unit: grouped k-fold, or leave-one-subject-out where the patient count permits. In R, this is implemented via group_vfold_cv() in rsample or groupKFold() in caret.The second most likely form of leakage is temporal leakage. Where the deployed model will predict forward in time, random partitioning of a series permits the algorithm to interpolate between observations it has effectively already seen.To effectively prevent this problem, blocked or rolling-origin designs preserve temporal ordering and produce an estimate corresponding to the deployed task (Roberts et al, 2017).The choice of performance metric is highly scrutinised. R^2 has no natural definition for a binary outcome in the sense familiar from OLS; pseudo-R^2 variants exist but do not address discrimination directly. For binary SaMD outputs, the Area Under the Receiver Operating Characteristic Curve (AUC-ROC) is standard. It is often said that AUC-ROC is misleading under severe class imbalance, and the claim requires care. AUC-ROC is computed from the true-positive and false-positive rates, both conditioned on the true class, and is therefore invariant to prevalence. The genuine difficulty is that a respectable AUC-ROC is entirely compatible with poor positive predictive value when the event is rare. This is because a small false-positive rate applied to a large population of negatives generates a great many false alarms. The Area Under the Precision-Recall Curve (AUC-PR) reflects this directly and is the more informative summary of the deployed decision (Saito & Rehmsmeier, 2015). Though, like the Brier score, AUC-PR depends on prevalence and is not comparable across populations with differing event rates. Where the clinical consequence of a false positive differs materially from that of a false negative, net benefit and decision curve analysis address the question more directly (Vickers & Elkin, 2006).The Brier score, a strictly proper scoring rule, summarises the overall accuracy of predicted probabilities. Under the Murphy decomposition, it partitions into a reliability (calibration) term, a resolution (discrimination) term entering with a negative sign, and an irreducible uncertainty term determined by the base rate. Because it is a summary measure, a low Brier score does not guarantee good calibration: a model can attain a respectable score through strong discrimination while remaining poorly calibrated. Calibration should therefore be verified directly, using a calibration plot together with the calibration slope and intercept.While nested cross-validation provides a solid estimate of internal model performance, it is not a substitute for external clinical validation. Regulators expect final out-of-sample (OOS) performance derived from test data held strictly independent of both training and tuning, and drawn so as to represent the intended use population. Temporal separation is one means of achieving this; geographical, site-level, and device-level separation address the generalisability question at least as directly. Performance should in addition be reported across clinically relevant subgroups rather than in aggregate alone, since an acceptable pooled estimate can conceal materially degraded performance in a subpopulation the device is indicated to serve. Finally, the reporting itself is now standardised: TRIPOD+AI (Collins et al., 2024) sets out what a prediction model study should disclose, and aligning the Statistical Analysis Plan (SAP) and clinical evaluation report to this checklist materially reduces the volume of reviewer questions.Classification and Regression Trees (CART) and Random ForestsFor the complex, non-linear interactions common in physiological data, tree-based methods offer powerful alternatives. Classification and Regression Trees (CART) partition the feature space into rectangular regions then make predictions based on the mean or mode of the observations within each terminal node. A single CART model is highly interpretable. The tree structure explicitly shows the decision rules, which is a significant advantage for regulatory submissions. Single trees, do however, suffer from high variance whereby small changes in the training data can drastically alter the tree structure.Random Forests mitigate this variance by aggregating (bagging) many trees. A Random Forest algorithm constructs B trees on bootstrapped samples, then considers a random subset of features at each split to decorrelate the trees. For classification, it predicts by majority vote. A critical output of Random Forests for SaMD validation is the Variable Importance Measure (VIM). In R’s randomForest package, the impurity-based importance for a classification model is reported as MeanDecreaseGini. This represents the total decrease in Gini impurity attributable to splits on a given variable, aggregated over all splits and averaged across the B trees. The analogous quantity for a regression forest is IncNodePurity, which is computed from the reduction in residual sum of squares.This impurity-based measure must be interpreted with care. It is biased toward continuous predictors and predictors with many categories (Strobl et al., 2007). Additionally, it becomes unreliable when predictors are strongly correlated. Under such correlation, importance may be diluted across a group of near-equivalent features. For regulatory feature attribution, permutation importance (MeanDecreaseAccuracy) is often preferable. This is the drop in out-of-bag accuracy when a variable’s values are randomly permuted. Two practical caveats should be noted here. First, conditional permutation importance is not implemented in randomForest; it requires fitting a conditional inference forest (via the party or permimp packages). Second, unrestricted permutation evaluates the model at feature combinations that never occur in the data, so the resulting importances are in part artefacts of extrapolation (Hooker et al., 2021). The defensible position is not that one measure is correct, but that the measure selected, its known biases, and the correlation structure of the predictors all must be stated in the regulatory submission.Providing these measures yields a defensible, non-parametric account of which physiological features are driving predictions on average. It is worth stating plainly what this does and does not deliver. Global variable importance is a population-level summary. It does not explain any individual prediction, and it is not equivalent to the algorithmic traceability that design controls and technical documentation are required to establish. Where the intended use requires a clinician to interrogate a particular output, per-prediction attribution is a separate undertaking, and in that setting the simpler, more auditable model may remain the better regulatory choice.From the ‘Locked Algorithm’ to the Predetermined Change Control PlanHistorically, ML-based SaMD was authorised as a “locked algorithm”: the model parameters were fixed at market authorisation, and any material subsequent changes triggered a fresh regulatory submission. This remains a valid path but is no longer the only one. The FDA’s 2024 guidance on Predetermined Change Control Plans (PCCPs) for AI-enabled device software functions allows a manufacturer to pre-authorise a defined envelope of future modifications. A PCCP comprises three elements: a Description of Modifications, a Modification Protocol, and an Impact Assessment. In October 2023, the FDA, Health Canada, and the MHRA jointly issued five guiding principles for such plans: that they be focused, risk-based, evidence-based, transparent, and lifecycle-oriented.In October 2023, the FDA, Health Canada, and the MHRA jointly issued five guiding principles for predetermined change control plans (PCCPs), holding that such plans should be 1. focused and bounded, 2. risk-based, 3. evidence-based, 4. transparent, and 5. oriented to the total product lifecycle. The FDA gave this approach concrete form in its 2024 final guidance on PCCPs for AI-enabled device software functions, which allows a manufacturer to pre-authorise a defined envelope of future modifications. A PCCP comprises three elements: a Description of Modifications, a Modification Protocol, and an Impact Assessment.The alternative option is a predetermined change control plan (PCCP), under which a defined envelope of future algorithm modifications is pre-defined at the time of the original regulatory submission. In October 2023, the FDA, Health Canada, and the MHRA jointly issued five guiding principles for such plans: that they be 1. focused and bounded, 2. risk-based, 3.evidence-based, 4.transparent, and 5. oriented to the total product lifecycle. The FDA’s 2024 guidance on PCCPs for AI-enabled device software functions then set the requirement for a PCCP to comprise of the following three elements: 1.a Description of Modifications, 2. a Modification Protocol, and 3. an Impact Assessment.The PCCP guidance does not stand alone. The FDA’s draft guidance of January 2025, Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations, addresses the total product lifecycle and speaks directly to matters within the biostatistician’s remit.These include data provenance, the independence of test data, performance evaluation, and subgroup reporting. If the device algorithm is governed by a PCCP, the SAP must define the performance boundaries of the deployed model and the criteria under which model retraining is permitted. Real-world device performance should then be monitored against those boundaries using control charts, such as a CUSUM on the prediction-error rate (Pardo, 2023). For certain devices the adjudicated outcomes must necessarily arrive too late to support that, or they are not available at a sufficient scale. In these cases, the PCCP must instead specify drift in the input and predicted-probability distributions as surrogates, together with the mechanism by which “ground truth” will periodically be recovered. Where actual drift falls outside the pre-authorised envelope, a Corrective and Preventive Action (CAPA) and a new regulatory submission are required.The EU MDR has no direct equivalent to PCCPs. The European analogue sits in the AI Act (Regulation (EU) 2024/1689). SaMD subject to third-party conformity assessment is high-risk under Article 6(1). Article 43(4) provides that, for high-risk systems that continue to learn after being placed on the market, changes pre-determined by the provider at the initial conformity assessment and documented in the technical file do not constitute a substantial modification. The resemblance to a PCCP is structural: a bounded, pre-specified, documented envelope of change. MDCG 2025-6 carries this across into device change control, taking the view that pre-determined changes falling within Article 43(4) should not be understood as a change to the certified device under MDR Annex IX Section 4.10. The guidance is non-binding, and notified bodies have so far applied it conservatively. The AI Act requirements are intended to be folded into the conformity assessment already conducted under MDR.The Digital Omnibus on AI, adopted as Regulation (EU) 2026/1744 and in force from 27 July 2026, defers the high-risk obligations for AI embedded in products regulated under Annex I (the route most SaMD follows) from 2 August 2027 to 2 August 2028. This deferral relieves timeline pressure without altering the substance of the data governance, logging, and post-market monitoring obligations required.ConclusionThe integration of machine learning into medical devices demands a higher level of vigilance. Regularisation, disciplined resampling, and global feature attribution are each necessary contributions to that vigilance. There is a regulatory obligation to ensure that predictive accuracy is established by validation in a population that resembles the one the device will meet and traceability is established by design controls and technical documentation, of which the statistical work forms a part. The biostatistician’s contribution is to ensure that the performance claim is honest, that the conditions under which it holds are stated, and that the mechanism by which it will be monitored over the device’s lifecycle is specified in advance.ReferencesBreiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984). Classification and Regression Trees. Wadsworth.Collins, G. S., et al. (2024). TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ, 385, e078378.Harrell, F. E. (2018). Road map for choosing between statistical modeling and machine learning. Statistical Thinking. https://www.fharrell.com/post/stat-ml/Hooker, G., Mentch, L., & Zhou, S. (2021). Unrestricted permutation forces extrapolation: variable importance requires at least one more model. Statistics and Computing, 31, 82.Meinshausen, N., & Bühlmann, P. (2010). Stability selection. Journal of the Royal Statistical Society: Series B, 72(4), 417–473.Pardo, S. A. (2023). Statistical Methods and Analyses for Medical Devices. Springer.Regulation (EU) 2024/1689 (Artificial Intelligence Act).Roberts, D. R., et al. (2017). Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure. Ecography, 40, 913–929.Saeb, S., Lonini, L., Jayaraman, A., Mohr, D. C., & Kording, K. P. (2017). The need to approximate the use-case in clinical machine learning. GigaScience, 6(5), 1–9.Saito, T., & Rehmsmeier, M. (2015). The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS ONE, 10(3), e0118432.Strobl, C., Boulesteix, A.-L., Zeileis, A., & Hothorn, T. (2007). Bias in random forest variable importance measures. BMC Bioinformatics, 8, 25.US Food and Drug Administration. (2024). Marketing Submission Recommendations for a Predetermined Change Control Plan for Artificial Intelligence-Enabled Device Software Functions.US Food and Drug Administration. (2025). Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations. Draft Guidance.van der Ploeg, T., Austin, P. C., & Steyerberg, E. W. (2014). Modern modelling techniques are data hungry. BMC Medical Research Methodology, 14, 137.Vickers, A. J., & Elkin, E. B. (2006). Decision curve analysis: a novel method for evaluating prediction models. Medical Decision Making, 26(6), 565–574.Zhao, P., & Yu, B. (2006). On model selection consistency of Lasso. Journal of Machine Learning Research, 7, 2541–2563.US FDA, Health Canada, & MHRA. (2023). Predetermined Change Control Plans for Machine Learning-Enabled Medical Devices: Guiding Principles.

Beneath the Surface of the Statistical Analysis PlanThere is an emerging commercial narrative that AI can automate the drafting of a Statistical Analysis Plan (SAP) directly from a clinical protocol. Extract the endpoints, populate a template, link to table shells, and the plan is ready. For anyone who has lived through the hours behind the process of drafting a SAP, the appeal is immediate.The promise of a shortcut, however, reveals as much about the one proposing it as it does about the task itself. When a company promises to automate the SAP, the question isn’t really about the technology. It’s about whether they understand the nature of the work at all.The SAP carries a study from database lock to regulatory submission. The reality of SAP development in medical device and IVD research is typically deeper, more iterative, and more technically demanding than any extraction-and-template workflow can capture. This is not a document that falls out of the protocol once the right fields are tagged. It is a crafted instrument that is negotiated, designed, and stress-tested. The gap between that reality and the automation promise is worth examining closely, especially for sponsors and early-career statisticians who may be tempted by the prospect of a faster path.The SAP Doesn’t Start with the Protocol AloneA protocol states the primary endpoint: “change from baseline in six-minute walk distance at 90 days.” But before a statistician can write the corresponding section of the SAP, they need the annotated eCRF. What exactly is collected? Is the distance recorded as a continuous value in metres, or is it captured as a categorical range? If it’s the latter, the analysis cannot proceed with the ANCOVA that was provisionally planned. The derivation of “change from baseline” also demands a precise baseline definition, such as “the last non-missing assessment on or before the procedure date”, and a visit windowing rule that selects the observation closest to Day 90, with ties broken by the later date. None of these operational details live in the protocol.Then there is the Data Management Plan, which specifies (for example) how missing and implausible values are queried, and that adverse events are coded using MedDRA version 29.x. That version number must be cited in the SAP, and the rules for treatment-emergent attribution must align with the cleaning conventions. If CDISC standards are in use, the SAP must name the exact ADaM variable that will hold the derived change-from-baseline, perhaps CHG in ADEFF at PARAMCD = "SIXMWD" and AVISIT = "Day 90", and map it to the SDTM source field. For an IVD study, the reference standard algorithm determines which subjects are true positives; a plan written without it may misclassify indeterminate results and inflate sensitivity.A tool that sees only the protocol cannot audit these connections. It cannot flag that the eCRF captures an ordinal pain scale when the SAP specifies a linear mixed model. It cannot notice that the imputation model must include the treatment arm to avoid bias toward the null. It is building on a partial foundation. The result is a document that may read smoothly but is silently disconnected from the data the study will later receive, a flaw that surfaces only when the programmer starts coding.The Integration Tax of a Thousand Micro-DecisionsTo be fair, artificial intelligence is incredibly capable of resolving an individual statistical quandary. If you ask an AI to define the Boolean logic for a treatment-emergent adverse event, or to calculate the design effect for a clustered sample size, it can often provide a highly competent sounding, localised response. AI vendors love to demonstrate these isolated feats of reasoning.It pays, however, to notice what those demonstrations conceal. Each question arrives already framed: someone has decided what counts as treatment-emergent, chosen the clustering unit, fixed the ICC. The AI is resolving a problem a human has already isolated. Writing a SAP is the opposite operation. Instead of answering questions one at a time, it involves a) deriving the specific bespoke questions that must be answered in this novel edge case (which represents the entire clinical investigation from an evidentiary and regulatory perspective); b) holding every relevant statistical resolution in view at once; and c) weighing them into a single, fully defensible conclusion that satisfies all relevant points of consideration.A SAP is not a single, easily-reconcilable problem. It is a composite of count-less minor considerations and micro-decisions, each of which must align with the others. Crucially, they do not align in a neat, linear sequence – they tend to conflict. A choice made to satisfy a clinical nuance might violate an operational constraint in the eCRF. A regulatory precedent might demand a conservative sensitivity analysis that causes the primary statistical model to fail to converge. The work is not executing a chain of dependent steps; it is holding a range of competing variables in tension simultaneously, weighing trade-offs across clinical, operational, and statistical domains until a single, defensible compromise is reached.There are some linear, cascading decisions which AI would currently struggle with in a regulated environment but could more-or-less handle with sufficient training and study-specific input documents. For example: A baseline variable definition dictates the corresponding change-from-baseline variable derivation; this derivation in turn dictates the missing data handling; the missing data handling dictates the imputation model; the imputation model dictates the table shells- and so forth. Even so, this represents one of a potential myriad of decision chains that comprise a SAP.When you account for the distributed nature of study sites, the variances in EDC builds, and the iterative estimand negotiations, the engineering required to orchestrate a system of agents capable of maintaining logical consistency across all competing input is far more complicated and energy intensive than a statistician simply drafting the document manually.The commercial pitch is that an AI generated draft, checked and revised by a human, nets out cheaper than a human-written one. This is a dubious assumption once the practicalities are considered. Verifying a SAP is not mere proofreading; it requires the formal, structured QC demanded by a highly regulated, high-commercial-stakes environment. That QC means independently re-deriving the document’s logic. A statistician reviewing their own draft carries the mental model that produced it: they know why the baseline definition took its form and what it constrains downstream. In independent QC, a statistician reviewing someone else’s draft must reconstruct that model from the text, decision by decision. Between two humans in the same field this is tractable, because the reviewer can generally assume the author reasoned their way to each decision, and an error tends to show up as a wrong step in that reasoning. With AI generated text the reviewer has less to lean on, and is closer to verifying each decision in its own right than to following a line of argument. Fluency makes this harder, not easier: confident, well-formed prose is exactly where fabricated detail and quietly incompatible decisions are hardest to spot. The drafting hours supposedly saved by AI are spent, with interest, on verification and subsequent re-drafting. The residual temptation to lighten that verification because a draft reads well is exactly the failure mode a GxP environment exists to prevent.The Plan is Forged in Conversation, Not just ExtractionA protocol might say: “Subjects in whom the device cannot be deployed will be replaced.” In a cross-functional kickoff, the statistician asks whether a deployment failure due to anatomical unsuitability is a screening issue, while a failure due to a fractured catheter is a device deficiency. The two scenarios lead to different analysis set assignments: one exclusion from mITT with careful documentation, the other an inclusion in the Safety Set and a composite strategy for the efficacy endpoint. The SAP must capture this logic as explicit Boolean conditions referencing eCRF fields. No protocol text spells it out.These discussions are where estimands are built under ICH E9(R1). For a peripheral atherectomy device, the intercurrent event of “bailout stenting” might be handled with a composite strategy: the subject is assigned a worst-case residual stenosis value, rather than being excluded. The five estimand attributes: (the treatment condition, the population, the variable, the handling of intercurrent events, and the population-level summary) must be stated for each endpoint. For a key secondary endpoint evaluated at six months, the population might be the mITT set, the variable the patient-reported VAS pain score, the intercurrent event of device explant handled by a hypothetical strategy, and the summary measure the difference in least-squares means from a mixed model with Kenward–Roger degrees of freedom.Such choices do not emerge from a template. They tend to emerge from the statistician playing devil’s advocate: “If the wound hasn’t healed by Day 30, and the patient stops attending, is that missing at random, or missing not at random?” That question determines whether the primary analysis uses multiple imputation under MAR, or whether a tipping-point sensitivity analysis must be pre-specified with a grid of delta shifts. The SAP is the record of that reasoning, but the reasoning itself happens in whiteboard sessions, not in document extraction.Every Study Carries Its Own FingerprintIn MedTech, while clinical trials often repeat a relative pattern, they do so with a good dose of individual nuance and some novelty thrown in. One study might involve clustered lesions within patients, requiring a generalised linear mixed model with a random patient intercept and an intra-class correlation coefficient (ICC) justified from literature. The sample size calculation must inflate the naïve estimate by the design effect (1 + (m-1)ρ), where m is the average cluster size and ρ is the ICC – an adjustment an automated power calculation might overlook. Another might be a multi-reader multi-case study whose variance must generalise across readers and cases alike, not a simple DeLong AUC comparison.For a Bayesian adaptive design borrowing strength from a previous device generation, the SAP must specify the power prior, the discounting parameter, and the simulation plan that demonstrates operating characteristics. The prior itself must be defensible to a Notified Body reviewing under MDR Annex XIV, an expectation that is context-specific and not codified in any public training corpus. A third trial might involve a safety composite endpoint like MACE, where the Boolean logic must adjudicate cardiovascular death, target-vessel myocardial infarction, and clinically driven revascularisation using data from an independent Clinical Events Committee, not site-reported terms.Each of these requires technical specifications that no off-the-shelf template can supply. The SAP must name the SAS procedure (PROC MIXED with ddfm=KR), the R package (lme4 with lmerTest, using ddf = "Kenward-Roger"), the confidence interval method for sensitivity (Clopper-Pearson because the numerator is small), and the pre-specified convergence fallback: if the unstructured covariance matrix fails, simplify to compound symmetry and document the change. An automation that fills in a generic “mixed model” or “appropriate nonparametric test” is writing a placeholder, not a plan. And in a document that must survive regulatory scrutiny, a placeholder is a liability.The SAP Is Read Under Pressure, Not at LeisureA SAP is an operational manual for a programmer who must translate each sentence into production code, and for the statistician who will later interpret the outputs and draft the clinical evaluation report. In MedTech, statistical programming is rarely straightforward. It often involves codifying highly sophisticated models and simulations that cannot be derived from the protocol alone, nor readily pieced together from a combination of input documents unless the SAP has explicitly synthesised them alongside the conversational decisions and considerations. Programmers and statisticians use the SAP as a workflow document. If they did not draft it themselves, it must be painstakingly clear about the methodological nuances and exactly how and why they play out practically. Both are working to deadlines, holding multiple rules in mind. A document that forces them to piece together the population definition from Section 3, the covariates from Section 4, and the missing data rule from Section 5 is a document that invites error and inefficiency at the QC stage, never mind the programming and analysis stages. Calibration is the craft: restate everything and the document bloats until nothing stands out; cross-reference everything and the reader assembles each analysis from fragments. Deciding what to repeat, where, and in how much detail is a delicate set of judgements in itself.A proponent of automation might counter this by suggesting that if the SAP is this precisely specified, the statistical programming itself could simply be automated next. If the exact specifications are in the SAP, and the analysis is exactly specified in the code, why not let an AI write the R or SAS scripts? The lived experience of statistical programming tells a different story. Certainly, AI has a role to serve as an efficiency booster in any coding context. This is, however, miles away from any kind of automation. The statistical programming process for a clinical study usually ends up unearthing as much nuance as the SAP draft and study design specification themselves. Writing the code forces a granular confrontation with the actual data- revealing undocumented edge cases, unexpected structural quirks, or logical paradoxes that remained invisible at the planning level. The code is not a mechanical translation of the SAP; it is the final, practical stress-test of the statistical logic. Translating a complex MACE composite or a Bayesian adaptive simulation into executable code requires a continuous stream of micro-decisions that loop right back to the original study design conversations. Automating the programming doesn’t eliminate the nuance; it just moves it one step further down the chain, where an AI is potentially even less equipped to resolve it.Why Capturing the Conversations Doesn’t Close the GapA natural response to the points above is to ask whether the missing conversations could simply be fed into the automation as well. After all, tools now exist (such as Aimanuensis Clinical Trials) that document and summarise clinical trial meetings – capturing every word of the kickoff discussions, the estimand negotiations, and the whiteboard debates. The protocol, eCRF, DMP, SDTM specs, plus the meeting transcripts: surely that all bridges the gap?In practice, it doesn’t. One reason lies in the non-linear nature of the conversations themselves. Clinical meetings are not linear briefings; they are iterative, exploratory, and unavoidably messy. The other lies in what the transcript would demand of its reader: not the resolution of any single question, but the same multi-front integration the meeting itself performed – weighing each stated preference against the eCRF structure, the estimand strategy, the regulatory precedent, and every decision already provisionally made. Dedicated software can extract decisions and action items from individual meetings. That is different to the kind of synthesis required for SAP development.A clinical meeting about a device study loops. A statistician raises a concern about an endpoint definition; the clinician pushes back with a real-world example; a regulatory colleague mentions a Notified Body precedent; the statistician revises the proposed handling and sketches a sensitivity analysis. Ten minutes later, the discussion circles back after a new point about the eCRF structure casts the earlier decision in a different light. The final agreed approach emerges not from any single documented decision, but from the synthesis of the entire exchange.Handing an AI a list of “decisions” doesn’t solve the integration problem. Decisions made on Monday are often invalidated by a competing practical consideration discovered on Wednesday. The SAP is the final, synthesised resolution of a constantly shifting web of clinical, operational, and regulatory constraints. An AI can be fed the individual pieces from a repository of decisions, but it cannot independently or reliably weigh competing considerations, resolve ambiguities, or fill in the gaps that the discussion itself never fully closed.Many of the most important resolutions happen between meetings, in the statistician’s own work. The quiet afternoon spent checking whether the proposed mixed model will converge with 40 subjects and an average cluster size of 1.2 – these insights are not captured in any meeting transcript until they are discussed after-the-fact. They are the result of solitary technical reflection, and they feed back into the next discussion, altering its course.Room for Tools, Space for CraftWhen a company offers to turn a protocol into a SAP at the push of a button, the deeper question is not about the algorithm. It’s about what they think the SAP is. The assumption baked into such tools is that the SAP is a derivative document, obtained by mapping protocol sections to plan sections to some standard statistical boilerplate.Ultimately, a SAP is the product of well-considered decisions made analytically by the biostatistician, balancing competing practical considerations. Once all the input documents and talking points have been digested, the proposed methodological approaches must be simulated and validated to confirm they hold up under the study’s specific constraints. An AI can draft a statistical method, but it cannot balance those competing considerations, nor can it independently design, run, and interpret the simulations required to prove the method is empirically sound. The final SAP is not just a written plan; it is an empirically tested engine that powers the study’s evidence gathering.For sponsors, the risk of trusting automation is a SAP that passes superficial review but contains methodological gaps, gaps a Notified Body may identify, or that surface only during the CER phase when rework is expensive. For early-career statisticians, the risk is absorbing a model of the work that mistakes document assembly for statistical design.None of this argues against automation. Intelligently extracting decisions and action items from clinical trial meeting transcripts, such as in the case of Aimanuensis, brings real efficiency. Tools with the ability to cross-check endpoint consistency between protocol and SAP, flag missing sections, or generate first-draft table shells aligned with the SAP’s population definitions would lighten the administrative layer of the work and be very much welcomed.In medical device and IVD research, where nearly every study deviates from the standard template, human judgement is not a bottleneck to be engineered away. It is the very thing that gives the SAP its authority. While removing human judgement might be a good idea for self-driving cars – writing a SAP is not a task that should be completed on auto-pilot with no recollection of the journey that just took place. Those suggesting otherwise are selling a shortcut to a destination they haven’t fully mapped.https://anatomisebiostats.co/wp-content/uploads/2026/07/MedTech_SAP_Handbook.pdf

Series: Advanced Biostatistics for MedTech: Bridging Clinical Evaluation and EngineeringThe Complexity of Device VariabilityUnder ISO 13485:2016 and MDR 2017/745 in the EU, medical device manufacturers must carefully control and validate their product manufacturing processes. A critical aspect of this validation is quantifying the sources of variability in the device’s Critical Quality Attributes (CQAs). While pharmaceutical manufacturing controls unit-to-unit variation through content-uniformity and dissolution testing, inherent variability in materials, machining, and assembly means that no two medical devices are perfectly identical.When a device is tested across multiple sites, operators, or lots, the observed variability is a composite of multiple sources. Standard fixed-effects ANOVA is meant to test differences between means rather than to partition variance, and classical variance component estimators become ambiguous under unbalanced conditions. This requires Variance Components Analysis (VCA) using Restricted Maximum Likelihood (REML) estimation, implemented via mixed-effects models. For a medtech biostatistician, VCA is not merely an academic exercise; it is the mathematical backbone of Measurement Systems Analysis (MSA), process validation, and the statistical defence of multi-site clinical investigations.Beyond the Randomized Block: REML for Variance PartitioningConsider a multi-site clinical or analytical study for a new continuous glucose monitor (CGM). The study involves 3 clinical sites, 5 operators per site, and multiple sensor lots. The measurement error (difference between the CGM and a reference method) is influenced by the site, the operator, the lot, and inherent sensor variability.A standard fixed-effects ANOVA is inappropriate here because the levels of “Site” and “Operator” are random samples from a larger population of possible sites and operators. The objective is to estimate the variance component attributable to “Site” in general, rather than to estimate the specific bias of “Site A”.The mixed-effects model is defined as:Y_{ijk} = \mu + S_i + O_{j(i)} + L_k + \epsilon_{ijk}Where:Y_{ijk} is the measurement error for lot k, operator j, site i.\mu is the grand mean (fixed effect).S_i \sim \mathcal{N}(0, \sigma_S^2) is the random effect of Site.O_{j(i)} \sim \mathcal{N}(0, \sigma_O^2) is the random effect of Operator nested within Site.L_k \sim \mathcal{N}(0, \sigma_L^2) is the random effect of Lot. \epsilon_{ijk} \sim \mathcal{N}(0, \sigma_\epsilon^2) is the residual error.Note that Lot is modelled as crossed with Site ( L_k ). This assumes the same sensor lots were evaluated across all participating sites. If instead each site received distinct lots, then Lot would need to be nested within Site ( L_{k(i)} ).In matrix notation, the model is defined as y = X\beta + Zu + \epsilon, where X is the design matrix for fixed effects, Z is the design matrix for random effects, and u \sim \mathcal{N}(0, G). The variance of the response vector is V = ZGZ^T + R, where R is the covariance matrix of the residuals. The goal of VCA is to estimate the elements of G (the variance components) and R.The Case for REML Over Maximum Likelihood (ML)In standard Maximum Likelihood (ML) estimation, the likelihood function is maximised for all parameters simultaneously.This includes the variance components. ML estimates of variance components are biased downward because they don’t account for the degrees of freedom lost in estimating the fixed effects (\beta). In small samples, common in device validation, this bias can cause underestimation of the true lot-to-lot or operator-to-operator variance, leading to a falsely optimistic conclusion about process control.Restricted Maximum Likelihood (REML) solves this by first applying a transformation to the data to eliminate the fixed effects (Patterson and Thompson, 1971). REML maximises the likelihood of the residual data, which in turn yeilds less-biased variance estimates (and yeilds unbiased estimates for balanced designs, provided the solution lies in the interior of the parameter space).Mathematically, REML maximises the likelihood of a set of error contrasts. These are linear combinations of the data whose distribution does not depend on the fixed effects. If K is a matrix such that K^T X = 0, then K^T y is a vector of error contrasts. The REML log-likelihood is based on the marginal distribution of these contrasts, which depends only on the variance components. This separates the estimation of variance from the estimation of the fixed effects, which in turn eliminates the downward bias. In R, this can be implemented using the lmer() function in the lme4 package:model <- lmer(Error ~ 1 + (1|Site) + (1|Site:Operator) + (1|Lot), data=df)Note the 1 + explicitly specifying the intercept. To specify the nested structure correctly without double-counting the site variance, the clean syntax is (1|Site:Operator). Note that (1|Site/Operator) expands to exactly (1|Site) + (1|Site:Operator), so it shouldn’t be added to a model that already contains (1|Site).Handling Unbalanced Data in VCAA significant advantage of mixed-effects models over traditional ANOVA (which relies on Method of Moments and expected mean squares) is the ability to handle unbalanced data gracefully. In multi-site device studies, it is common for a site to drop a patient, an operator to leave the study, or a lot to have fewer sensors.Traditional ANOVA sums of squares become mathematically ambiguous and highly sensitive to missingness when data are unbalanced. Depending on whether Type I, II, or III sums of squares are used, the variance attribution can change dramatically, leading to unresolvable debates during regulatory reviews. Additionally, the Method of Moments can occasionally yield negative variance estimates, which are a legitimate output of estimating equations that don’t constrain the parameter space but are practically nonsensical.REML, however, uses an iterative optimisation algorithm (such as the Expectation-Maximisation or Newton-Raphson algorithm) to find the variance components that maximise the likelihood of the observed data, regardless of balance. By operating on the likelihood function rather than sums of squares, REML gracefully handles missing cells and naturally constrains variance estimates to be non-negative. This robustness makes REML the preferred approach for VCA in medical device validation.The Gauge R&R Connection: MSA Under ISO 13485Variance Components Analysis is the mathematical backbone of Measurement Systems Analysis (MSA) and supports compliance with ISO 13485 (Clause 7.6, Control of monitoring and measuring equipment) for process and measurement system validation.In this CGM example, the response variable is the measurement error: the difference between the CGM and a reference method. As the response is already an error term, the total variance ( \sigma_{total}^2 ) represents total error variance rather than total product variation. The variance components estimated by the mixed-effects model ( \sigma_S^2, \sigma_O^2, \sigma_L^2, \sigma_\epsilon^2 ) partition the sources of measurement error rather than traditional AIAG Gauge R&R components.A Notified Body expects total measurement error to be acceptably small relative to the clinical or analytical specification limits. If the operator component ( \sigma_O^2 ) dominates, ie. the device’s performance is overly sensitive to user technique, this indicates a need for improved training or design changes. If the site component ( \sigma_S^2 ) dominates, it suggests systematic differences in how the device performs across different clinical environments (e.g., due to local sample handling or environmental factors). If the residual variance ( \sigma_\epsilon^2 ) dominates, the device itself lacks the necessary analytical precision.Note that standard AIAG R&R metrics like the Number of Distinct Categories (ndc) and the Study Variation ratio ( \sigma_{MS} / \sigma_{total} ) require true part-to-part variation ( \sigma_{PV} ) in the response. As this model analyses measurement error, part variation is excluded by design, which makes those specific AIAG metrics invalid for this response type. Instead, total error variance ( \sigma_{total}^2 = \sigma_S^2 + \sigma_O^2 + \sigma_L^2 + \sigma_\epsilon^2 ) should be compared directly against the allowable error margins (e.g., ISO 15197 accuracy criteria for CGMs). This sufficiently discriminates for process validation.Nested vs. Crossed Designs: The Pseudoreplication TrapA common error in statistical analysis plans is mis-specifying a design’s nesting structure. In the CGM example, Operator 1 at Site A is a different person than Operator 1 at Site B. Therefore, Operator is nested within Site, specified as (1|Site:Operator). If the same operators tested the device at all sites (a rare logistical feat), the design would be crossed and specified as (1|Site) + (1|Operator).Failing to specify the nesting structure correctly leads to pseudoreplication (treating non-independent observations as independent) and severely underestimated standard errors. Whether Operator 1 at Site A is the same person as Operator 1 at Site B is a fact about the study design. It is not recoverable from the data frame itself. It is implicitly nested data with reused labels and looks byte-identical to a fully crossed design. The practical safeguard against pseudoreplication is to relabel operators uniquely (e.g., creating composite IDs like SiteA_Operator1) before fitting the model. This makes implicit nesting explicit and immune to mis-specification. The experimental design must be carefully mapped to the lmer() syntax. Crossed random effects estimate separate variance components for each factor when every level of one factor co-occurs with every level of the other. In contrast, nested random effects partition the variance hierarchically.Testing Variance Components: The Boundary ProblemWhile REML provides point estimates for variance components, regulatory submissions often require a formal hypothesis test to determine if a variance component is statistically significant. For example, is the lot-to-lot variance significantly greater than zero?The standard approach is a Likelihood Ratio Test (LRT) comparing the full model with the random effect to a reduced model without it. The null hypothesis (variance = 0) lies on the boundary of the parameter space (variances cannot be negative), thus the standard chi-square asymptotic distribution is incorrect.Under standard conditions, -2 times the log-likelihood ratio follows a \chi^2 distribution. Under the boundary condition, the correct reference distribution is a 50:50 mixture of \chi^2_0 (a point mass at zero) and \chi^2_1 distributions (Self & Liang, 1987). In R, base functions like anova() will output a standard p-value based on the naive \chi^2_1. This is technically conservative: it roughly doubles the correct p-value, reducing power and inflating the Type II error rate. This can lead to failure to detect a real lot-to-lot variance and falsely conclude the variance is negligible. The better solution is to use simulation-based tests specifically designed for boundary conditions. The asymptotic 50:50 mixture provides a simple analytical correction using simple halving. The exactRLRT function from the RLRsim package uses exact finite-sample simulation for models with a single variance component, rather than simple halving (Crainiceanu & Ruppert, 2004). This makes the latter the preferred approach in small samples to recover the correct, more powerful p-value.From VCA to Statistical TolerancingThe ultimate purpose of VCA in device design is not merely to pass a Gauge R&R audit, but to inform statistical tolerancing. Once the variance components (\sigma_S^2, \sigma_O^2, \sigma_L^2, \sigma_\epsilon^2) are quantified, they sum to predict the total variance of the device in the field:\sigma_{total}^2 = \sigma_S^2 + \sigma_O^2 + \sigma_L^2 + \sigma_\epsilon^2Taking the square root yields the total standard deviation (\sigma_{total}), which is the critical input for statistical tolerancing. Because \sigma_{total} aggregates across sites, operators, and lots, it represents long-term, overall variation. By AIAG SPC convention, a capability index built on overall sigma is technically the Process Performance Index (Ppk), which penalises any off-centring of the process mean:Ppk = \min\left( \frac{USL - \mu}{3\sigma_{total}}, \frac{\mu - LSL}{3\sigma_{total}} \right)Note that for this CGM example, the measurement-system components (\sigma_\epsilon^2, \sigma_O^2) are included in the total variance because the device’s CQA is itself a measurement output. For a physical dimension, the gauge variance would typically be partitioned out to assess the true manufacturing process capability independently of the measurement noise. Sensitivity analysis can be performed by decomposing the variance. If \sigma_O^2 dominates, tightening the manufacturing tolerance on the device itself will not improve Ppk; only operator training will. This data-driven approach to design optimisation is what ISO 13485 and MDR Annex I (General Safety and Performance Requirements) expect to justify the manufacturing control and benefit-risk profile of a device.Variance Components Analysis is indispensable for medical device validation. REML and mixed-effects models can appropriately partition variability across sites, operators, and lots. This supports compliance with ISO 13485 Clause 7.6 and produces actionable insights for design optimisation. The correct specification of nested versus crossed designs, the use of boundary-adjusted likelihood ratio tests, and the translation of VCA into statistical tolerancing are essential for a defensible statistical submission.ReferencesPardo, S. A. (2023). Statistical Methods and Analyses for Medical Devices. Springer.International Organization for Standardization. (2016). ISO 13485:2016 – Medical devices – Quality management systems.Automotive Industry Action Group (AIAG). (2010). Measurement Systems Analysis (MSA) Reference Manual (4th ed.). AIAG.Searle, S. R., Casella, G., & McCulloch, C. E. (1992). Variance Components. John Wiley & Sons.Pinheiro, J. C., & Bates, D. M. (2000). Mixed-Effects Models in S and S-PLUS. Springer.Patterson, H. D., & Thompson, R. (1971). Recovery of inter-block information when block sizes are unequal. Biometrika, 58(3), 545-554.Self, S. G., & Liang, K. Y. (1987). Asymptotic properties of maximum likelihood estimators and likelihood ratio tests under nonstandard conditions. Journal of the American Statistical Association, 82(398), 605-610.Crainiceanu, C. M., & Ruppert, D. (2004). Likelihood ratio tests in linear mixed models with one variance component. Journal of the Royal Statistical Society: Series B, 66(1), 165-185.

Series: Advanced Biostatistics for MedTech: Bridging Clinical Evaluation and EngineeringThe Statistical Advantage of Iterative Medical Device DevelopmentMedical devices are iterative, physical, and engineerable. By the time a medical device reaches a pivotal clinical investigation, it has undergone extensive Verification and Validation (V&V) bench testing, biocompatibility assessments, and often animal studies. The physics of the device (such as its tensile strength, fatigue life, thermal dissipation, or sensor accuracy) are at this point quantitatively well-characterised as random variables with well-estimated expected values and variances.This robust pre-clinical data provides a sound foundation for Bayesian informative priors. When designing the clinical trial, the biostatistician can mathematically encode this engineering data into a Bayesian framework. The goal being the construction of adaptive clinical investigations that are statistically powered to detect treatment effects while having the flexibility to adapt to interim data. This approach is potentially advantageous under the constraints of the EU Medical Device Regulation (MDR 2017/745) and the UK MHRA requirements for clinical investigations.The Regulatory Precedent: FDA CDRH and the EU MDRThe FDA Center for Devices and Radiological Health (CDRH) issued its Guidance for the Use of Bayesian Statistics in Medical Device Clinical Trials in 2010, explicitly endorsing Bayesian adaptive designs for premarket approval (PMA) and 510(k) submissions.In Europe, while the MDR does not mandate a specific statistical approach, Article 62 (Clinical Investigations) and Annex XV require that the investigation plan minimise risk and burden to subjects while generating robust evidence. Bayesian adaptive designs are well-suited to this regulatory imperative for obvious reasons. Namely, they allow studies to stop early for efficacy or futility, thereby minimising patient exposure to potentially suboptimal treatments. Notified Bodies and the MHRA are increasingly receptive to these designs, provided the Statistical Analysis Plan (SAP) solidly defends the prior distributions used and the operating characteristics (Type I error and power) of the adaptive algorithm.Constructing Informative Priors from Engineering DataThe derivation of the prior distribution for the clinical parameter of interest, \theta_{clin} (e.g., a clinical treatment effect or in-vivo success rate) is the cornerstone of a Bayesian study design.Bayes’ theorem states that the posterior distribution is proportional to the product of the prior distribution and the likelihood function of the observed data:f_{posterior}(\theta_{clin} \mid y) \propto L(y \mid \theta_{clin}) \cdot f_{prior}(\theta_{clin})The Meta-Analytic-Predictive (MAP) prior is the gold standard for pooling historical clinical data. It assumes exchangeability between historical and new clinical studies. Applying a MAP prior directly to engineering V&V bench data is a statistical fallacy. Bench data (e.g., analytical accuracy in a control solution) and clinical data (e.g., in-vivo accuracy in human tissue) are not noisy, exchangeable measurements of the same parameter.To bridge the gap, the biostatistician can use a commensurate prior framework (or a heavily discounted power prior). This approach explicitly models the “bench-to-human gap” rather than ignoring it. Let \theta_{bench} be the parameter estimated from V&V data, and \theta_{clin} be the clinical parameter. We model the clinical prior as the bench estimate shifted by an expected bench-to-clinical bias and widened by our uncertainty about that shift:\theta_{clin} \sim N(\theta_{bench} - b_{gap}, \tau^2_{gap})Where b_{gap} is the anticipated degradation in performance moving from bench conditions to human physiology, and \tau^2_{gap} quantifies uncertainty about its magnitude. If \tau^2_{gap} is set small, we are asserting that the bench data closely predicts the clinical outcome; if set large, we express scepticism. Setting b_{gap} = 0 asserts that bench conditions are unbiased for in-vivo performance, which for most measurement technologies is optimistic. This is because analytical accuracy in a control solution is systematically better than in-vivo accuracy in tissue. Both b_{gap} and \tau^2_{gap} must be justified in the SAP from whatever bridging evidence exists, such as animal studies, prior-generation clinical data, or documented matrix-effect testing. Where no bridging evidence exists, this is an argument for using a weakly informative prior rather than a discounted informative one. The sign convention here assumes higher values of \theta denote better performance. For error rates, failure rates, or other parameters where lower is better, the bias term is added rather than subtracted: \theta_{clin} \sim N(\theta_{bench} + b_{gap}, \tau^2_{gap}). In a true Hobbs commensurate prior, this commensurability parameter carries its own hyperprior and is estimated from the data, so borrowing adapts dynamically if prior-data conflict arises. If \tau^2_{gap} is instead pre-specified as a fixed constant, the design relies on fixed borrowing (closer to a power prior) rather than a fully adaptive commensurate prior.To account for the inherent risk that human physiology fundamentally contradicts the engineering data (e.g., an unforeseen biological interferent), this prior must be robustified. A robust prior mixes the informative commensurate prior with a vague, heavy-tailed distribution (e.g., a Student-t distribution with low degrees of freedom):\pi_{robust}(\theta_{clin}) = (1 - w) \cdot \pi_{commensurate}(\theta_{clin}) + w \cdot \pi_{vague}(\theta_{clin})Where w is the mixing weight on the vague component (e.g., 0.1 to 0.5). Larger values of w buy more robust protection. This ensures that if the clinical data fundamentally contradicts the engineering data, the heavy-tailed component allows the posterior to “escape” the overly optimistic engineering prior. Documenting the derivation of \tau^2_{gap}, b_{gap}, and w is a critical regulatory requirement.Predictive Probability and Adaptive Stopping in Clinical InvestigationsThe primary advantage of using the Bayesian framework in clinical investigations is the ability to compute the Predictive Probability of trial success at interim analyses. This allows for adaptive stopping rules that do not rely on the pre-specified alpha-spending functions required by frequentist group sequential designs (e.g., O’Brien-Fleming boundaries). Bayesian adaptive designs do, however, substitute one form of pre-specification for another rather than reducing these demands. Namely, the interim timings, thresholds, and simulations demonstrating Type I error control must still be locked in the protocol and SAP.Let \delta represent the clinical treatment effect. Study success is defined as the posterior probability of \delta exceeding a pre-specified margin crossing a threshold 1 - \epsilon:P(\delta > \delta_{margin} \mid y) \geq 1 - \epsilonAt an interim analysis with data y_{interim}, we calculate the predictive probability of the device eventually crossing the success threshold if the trial were to continue to its maximum sample size N. This requires integrating over the distribution of future datay_{future}:PP_{success} = \int P(\text{Success} \mid y_{interim}, y_{future}) \cdot P(y_{future} \mid y_{interim}) \, dy_{future}The term P(y_{future} \mid y_{interim}) is the posterior predictive distribution. If PP_{success} falls below a pre-specified futility boundary (e.g., 0.10), the trial is stopped for futility. If PP_{success} exceeds an efficacy boundary (e.g., 0.95), the trial stops early for success.Note on convention: The SAP must explicitly specify whether this 0.95 efficacy boundary applies to the predictive probability (PP_{success}) or the current posterior probability P(\delta > \delta_{margin} \mid y_{interim}). Conventionally, early efficacy stopping keys off the current posterior probability, while predictive probability is the natural futility tool. Using predictive probability for both is defensible, but the two quantities do behave differently and must be clearly distinguished in the protocol.Controlling Type I Error and Operating CharacteristicsWhile Bayesian purists would argue that Type I error is a frequentist construct, FDA CDRH guidance and Notified Body expectations mandate that Bayesian adaptive designs demonstrate control of the frequentist Type I error rate.Adaptive stopping introduces multiplicity opportunities, therefore, the SAP must simulate the trial design under the null hypothesis (\theta = \theta_{margin}) to calculate the Bayesian Type I error rate:\alpha_{Bayesian} = P(\text{Stop for Success} \mid \theta = \theta_{margin})If \alpha_{Bayesian} exceeds the required threshold (typically 0.025 for a one-sided hypothesis, corresponding to a two-sided 0.05), the efficacy probability threshold must be calibrated upward (made more stringent, e.g., from 0.95 to 0.975 or 0.99) until the simulated Type I error is controlled. If calibration alone can’t recover Type I error control without rendering the trial infeasible, the informative prior itself must be weakened. Lowering the threshold would make success easier to declare under the null, thereby increasing the false-positive rate.The study design must demonstrate adequate power (1 - \beta). Power is the probability of correctly rejecting the null hypothesis when the alternative is true. The SAP must include Monte Carlo simulations proving the design has sufficient power at the stated minimum clinically important difference.Post-Market Surveillance as a Bayesian Updating ProcessA highly effective application of Bayesian statistics in medtech is Post-Market Surveillance (PMS). Under MDR Article 83, PMS is a continuous, proactive process. Periodic Safety Update Reports (PSURs), mandated under Article 86, must quantify the benefit-risk profile of the device using real-world data (RWD).Frequentist statistics offer robust tools for continuous PMS, such as CUSUM or EWMA control charts, which are explicitly designed to detect shifts in a process mean over time. These frequentist sequential methods struggle, however, to formally incorporate historical pre-market clinical data into the real-world monitoring phase. Bayesian inference, being inherently sequential, is well suited to continuous PMS because the posterior distribution from the pre-market clinical investigation can serve as the foundation for the post-market prior. Just as with the bench-to-clinical gap, in should be noted that a pre-market population is screened, protocol-managed, and treated by selected investigators, whereas real-world post-market use is none of those. This non-exchangeability must again be modelled explicitly. The post-market prior should therefore apply the same commensurate machinery: incorporating a bias shift and widened variance to account for the predictable degradation in performance moving from a controlled trial to real-world use.Let \lambda represent the device’s true failure rate. As real-world complaint data (y_{pms}) accrues monthly, the posterior is continuously updated:P(\lambda \mid y_{clinical}, y_{pms}) \propto L(y_{pms} \mid \lambda) \cdot P(\lambda \mid y_{clinical})If the regulatory action threshold is a failure rate exceeding \lambda_{action} (e.g., 2\%), the PMS plan triggers a Corrective and Preventive Action (CAPA) when the posterior probability of exceeding this threshold breaches a pre-defined limit:P(\lambda > \lambda_{action} \mid y_{clinical}, y_{pms}) \geq 0.80This framework allows manufacturers to distinguish between random noise and a true signal of device degradation. Evaluating a monthly fixed posterior threshold, however, still generates a cumulative false-alarm probability over time, much like frequentist control charts. Similar to pre-market adaptive designs, the PMS plan must simulate the operating characteristics of this sequential Bayesian rule under the null hypothesis (acceptable performance) to characterise its long-run false-alarm rate for regulatory reviewers.If a design change is implemented to address a CAPA, the prior can be discounted (e.g., using a power prior with an exponent a_0 < 1) to reflect that the post-modification device is no longer perfectly represented by the pre-modification historical data.Documenting the Bayesian DefenceTo survive regulatory scrutiny, a Bayesian SAP must be exhaustive. Briefly, the following elements are expected:Prior Justification: The SAP must detail the exact source of the engineering and historical data that was used to construct the informative prior. The rationale for the robustification weight w and the methodology for down-weighting historical data (if using power priors) must be explicitly defended against accusations of prior-optimism.Operating Characteristics: The SAP must include extensive Monte Carlo simulations demonstrating that the adaptive design controls Type I error (\alpha) across the null parameter space, and that it achieves the desired power (1 - \beta) at the minimum clinically important difference.Interim Analysis Plan: The specific timing of interim looks, the predictive/posterior probability thresholds for stopping, and the maximum sample size must be specified and locked in the protocol.Sensitivity Analysis: The SAP must include sensitivity analyses that will demonstrate how the posterior conclusions change if a vague, non-informative prior is used instead of the informative prior. If the conclusions flip, the trial design is highly prior-dependent and may be rejected.ConclusionBayesian adaptive designs offer a regulatorily accepted pathway to optimise medical device clinical investigations. By encoding robust engineering V&V data into informative priors, biostatisticians can design studies that are smaller and more flexible than frequentist alternatives. For MDR-mandated Post-Market Surveillance, the Bayesian framework allows for continuous, proactive benefit-risk evaluation. While it does not require a pre-specified frequentist alpha-spending function, it still incurs, and must be calibrated for, a long-run false-alarm cost from repeated testing.References:Pardo, S. A. (2023). Statistical Methods and Analyses for Medical Devices. Springer. [Chapters 4, 7, 11, 13]US Food and Drug Administration. (2010). Guidance for the Use of Bayesian Statistics in Medical Device Clinical Trials.European Parliament and Council. (2017). Regulation (EU) 2017/745 on medical devices (MDR), Articles 62, 83, 86, and Annex XV. Official Journal of the European Union.Hobbs, B. P., Carlin, B. P., Mandrekar, S. J., & Sargent, D. J. (2011). Hierarchical commensurate and power prior models for adaptive incorporation of historical information in clinical trials. Biometrics, 67(3), 1047-1056.Schmidli, H., Gsteiger, S., Roychoudhury, S., O’Hagan, A., Spiegelhalter, D., & Neuenschwander, B. (2014). Robust meta-analytic-predictive priors in clinical trials with historical control information. Biometrics, 70(4), 1023-1032.

Series: Advanced Biostatistics for MedTech: Bridging Clinical Evaluation and EngineeringThe Regulatory Rejection of “Industry Standard” Sample SizeOne of the most frequently cited Major Non-Conformities in Notified Body audit reports under the EU Medical Device Regulation (MDR 2017/745) and IVDR 2017/746 is the inadequate statistical justification of sample size estimates. During design control reviews and Clinical Evaluation Report (CER) appraisals, technical reviewers consistently reject sample size rationale predicated on “historical precedent” or “industry standard practice.”Justifying sample sizes for medical devices requires a fundamentally different statistical framework to pharmaceutical studies. The goal of medical device studies is not always to evaluate a clinical treatment effect. Other common objectives include demonstrating physical reliability, conformance to engineering specifications, or an acceptable probability of use error. A medical device technical documentation file is often heavily dominated by non-clinical Verification and Validation (V&V) data. This is especially true for line extensions or devices relying on substantial equivalence (510(k)) and equivalent device routes (MDR Article 61). In these cases V&V activities constitute the primary evidence of safety and performance, often without the need for a new clinical trial. V&V activities can consist of bench testing, biocompatibility, usability engineering (IEC 62366-1), and manufacturing validation. Even for entirely novel devices requiring pivotal clinical investigations, thorough V&V is a prerequisite gatekeeper.In sample size estimation for conformance V&V, standard power calculations for detecting a difference in means are replaced by power calculations to assess a limit, which are built around interval estimates. To construct a justifiable sample size estimation for a technical documentation file, the biostatistician must navigate a complex interplay between confidence, tolerance intervals, reliability assurance, and the probability of validation success (Pardo, 2023).The Triad of Statistical Intervals: Confidence, Prediction, and ToleranceThe root cause of statistical friction in device V&V is potentially the conflation of three distinct statistical intervals. Confidence Intervals (CIs) characterise the uncertainty around a population parameter (e.g., the mean). Pardo (2023) dissects Neyman’s confidence framework, emphasising that a 95% CI constructed from a sample of size n implies that if the experiment were repeated infinitely, 95% of the resulting intervals would contain the true population mean. A CI says nothing about the proportion of individual devices that will meet specification. V&V activities frequently require Prediction Intervals and/or Tolerance Intervals instead.A Prediction Interval addresses the next single observation. A 95% prediction interval provides bounds within which the next single randomly sampled unit will fall with 95% probability.A Tolerance Interval, governed by ISO 16269-6, provides bounds that contain a specified proportion ( p ) of the population with a specified confidence level ( \gamma ). When validating a physical dimension against a specification limit, a Notified Body expects evidence that virtually all individual units will conform, not just that the average unit conforms.For a normally distributed characteristic with unknown mean and variance, a one-sided tolerance bound is derived exactly using the non-central t-distribution. A two-sided tolerance interval has no exact closed form and is typically computed using a chi-square approximation (such as the Howe method), which is the basis for the tabulated factors in ISO 16269-6. For a two-sided interval, the bounds are constructed as:\bar{X} \pm k SWhere \bar{X} is the sample mean and S is the sample standard deviation.To state that a sample size of n=30 was chosen to “guarantee 95% confidence,” is mathematically meaningless without specifying what is being guaranteed. If the goal is to demonstrate that 95% of the population falls within the specification limits with 95% confidence (a 95/95 tolerance interval), the required sample size and the resulting interval width are substantially different from a 95% confidence interval for the mean. Biostatisticians must enforce this strict terminology in V&V protocols.The c=0 Acceptance Sampling LogicFor attribute data (pass/fail, go/no-go), sample size justification cannot rely on standard power curves. Instead, it relies on the operating characteristics of acceptance sampling plans. Under ISO 2859-1 and the Squeglia Zero Acceptance Number (c=0) framework, sample sizes are derived from the binomial probability distribution.In a c=0 plan, if a single defective unit is found in a sample of size n, the lot or batch fails the validation. The statistical rationale is framed around two parameters:P: The desired confidence level (e.g., 0.95) that the true defective rate is ≤ p₀.p_0: The maximum acceptable defective rate in the population. In this confidence framing,p0​ represents the consumer’s-risk point, often termed the Rejectable Quality Limit (RQL), Limiting Quality (LQ).Under a binomial model, the probability of accepting a lot (finding zero defectives in a sample of size n ) when the true population defective rate is p_0 is:P(\text{Accept}) = (1 - p_0)^nIf the objective is to be P confident that the lot has a defective rate no higher than p_0, the required sample size is derived algebraically:n = \text{trunc}\left[ \frac{\ln(1 - P)}{\ln(1 - p_0)} \right] + 1For example, to achieve 95% confidence (P = 0.95) that the true defective rate is less than 5% (p_0 = 0.05), the required sample size is n = 59. While p_0 is the consumer’s-risk point, it numerically coincides with the exact one-sided 95% upper confidence bound (e.g., 0 failures in 59 units yields an exact upper bound of 4.951%). This algebraic derivation is the absolute minimum defence required in a V&V protocol.The binomial derivation assumes an infinite population. For finite populations (e.g., a specific manufacturing batch of 500 units), the hypergeometric distribution is technically more accurate, though the binomial is often used as a convenient large‑lot approximation. For small lots the exact hypergeometric calculation should be used. The c=0 logic is entirely frequentist. It does not account for prior knowledge of the manufacturing process. For post-market surveillance or legacy medical device validations where historical batch data exists, Bayesian augmentation of the c=0 plan can significantly reduce the required sample size while still providing a high level of assurance.The prior must be well justified and sensitivity analyses used to confirm the conclusions are not unduly driven by optimistic assumptions.Variables Sampling Plans and the Non-Central t-DistributionFor continuous data (e.g., tensile strength, flow rate, assay precision), attribute sampling is statistically inefficient. Instead, Variables Sampling Plans based on the Process Capability Index (Cpk) are mandated.The hypothesis test is structured to demonstrate that the true population Cpk exceeds a target value K_0 (e.g., K_0 = 1.0 , corresponding to 3\sigma capability or 99.73% conformance):H_0: Cpk < K_0 \quad \text{vs.} \quad H_1: Cpk \geq K_0To demonstrate with 95% confidence that the true Cpk is at least 1.0, a Notified Body expects the lower 95% confidence bound of the sample Cpk to exceed 1.0. A common, yet fatal, flaw is to simply set the acceptance criterion at the target value (e.g., requiring the sample Cpk to be \geq 1.0). If the true population Cpk is exactly 1.0, the probability that the sample Cpk will exceed 1.0 is approximately 52.7%. Setting the bar at 1.0 provides essentially no regulatory assurance that the true Cpk is actually 1.0.To ensure the lower 95% confidence bound exceeds 1.0, the acceptance criterion must be raised. For a sample size of n = 36 and a target K_0 = 1.0, the required sample Cpk must be approximately 1.27. Under this stringent criterion, if the true Cpk is exactly 1.0, the probability of the sample passing the test is only 5% (the \alpha risk). This correctly places the statistical protection on the regulator/consumer, ensuring a marginally capable process will not easily pass.These critical values are derived using the non-central t-distribution, where the non-centrality parameter (\delta_{ncp}) is a function of the sample size and the target capability:\delta_{ncp} = 3 \sqrt{n} K_0Plotting the probability of passing against a range of true population Cpk values produces the operating characteristic curve for the plan; its inclusion in the SAP is the gold standard for defending continuous data sample sizes to a Notified Body.The Concept of Assurance (Bayesian Expected Power)A persistent flaw in frequentist sample size justification is the assumption that the true effect size (or variance, or defective rate) is known with certainty. In reality, the true parameters are estimates derived from small pilot studies, engineering tolerances, or literature reviews. If the true effect size is overestimated during planning, the study will be underpowered. If it is underestimated, the study is overpowered, wasting resources and potentially exposing subjects to unnecessary risk.The MDR Article 61 requires robust clinical evaluation that accounts for uncertainty, but it doesn’t mandate a specific statistical approach. Relying solely on standard frequentist power calculations, which assume the true effect size is known with certainty, can be a limitation. The advanced statistical solution to this limitation is the concept of Assurance (also known as Bayesian Expected Power or Probability of Success).Assurance integrates the uncertainty of the true parameter over its prior distribution. Instead of calculating power at a single point estimate, assurance calculates the expected power over the entire space of plausible parameter values.Let \theta be the true parameter of interest (e.g., the true mean difference, the true variance, or the true defective rate). Frequentist power is conditional on a specific value of \theta:\text{Power} = P(\text{Reject } H_0 \mid \theta)Assurance marginalises over the prior distribution \pi(\theta):\text{Assurance} = \int P(\text{Reject } H_0 \mid \theta) \pi(\theta) d\thetaFor medical devices, deriving the prior distribution \pi(\theta) is highly tractable. Device development is iterative; bench-testing data, computer simulations, and animal studies provide robust prior information regarding device performance. By formally translating this prior data into a distribution (e.g., a conjugate Beta distribution for a binomial defective rate, or a Normal-Inverse-Gamma for normal means and variances), the assurance calculation estimates more realistically the probability that the V&V activity will succeed.The strongest regulatory precedent for Bayesian methods in the medical device industry comes from the FDA/CDRH’s 2010 Guidance for the Use of Bayesian Statistics in Medical Device Clinical Trials. EU Notified Bodies are increasingly receptive to these approaches when they transparently incorporate the totality of evidence rather than relying on arbitrary point estimates. If assurance is used, the SAP must detail the elicitation of the prior distribution and conduct sensitivity analyses to demonstrate that the calculation is not unduly influenced by overly optimistic prior assumptions.Reverse-Engineering Sample Sizes: The Engineering ConstraintIn the medtech industry, the biostatistician frequently encounters a scenario where the sample size is dictated by physical, ethical, or economic constraints rather than statistical optimisation. For example:A wear simulator may only have 6 test stations.Only 10 animal subjects are ethically justifiable for a chronic toxicity study.The budget allows for the destruction of only 15 units for fatigue testing.When the sample size nis fixed by external constraints, the statistical paradigm shifts from “What n is required to achieve 95% power?” to “What statistical assurance can be defensibly claimed with this fixed n?”This requires reverse-engineering of the statistical parameters. Using the c=0 binomial logic, if n=10 is the absolute maximum, and 95% confidence (P=0.95) is required, the maximum defective rate p_0 that can be claimed is derived by rearranging the binomial formula:p_0 = 1 - (1 - P)^{1/n}For n=10 and P=0.95, the claimable p_0 is approximately 0.259 (25.9%). If the engineering team requires a claim of 5% defective rate, n=10 is demonstrably insufficient. The biostatistician must clearly communicate this gap between engineering ambition and statistical reality.For continuous data with a fixed n, reverse-engineering requires calculating the precision of the estimate. If n=15, the width of the 95/95 tolerance interval will be wide. The biostatistician must calculate the expected width and determine if the tolerance interval will realistically fit within the engineering specification limits. If the specification limits are narrow, n=15 will yield a tolerance interval that exceeds the limits, resulting in a validation failure.Presenting a “Reverse-Engineered Assurance Analysis” in the technical documentation file demonstrates to the Notified Body that the manufacturer understands the statistical limitations of their study design. It shifts the regulatory conversation from “Why did you choose this sample size?” to “Given this sample size, here is the maximum statistical risk we are assuming,” which is then formally documented in the Risk Management File (ISO 14971:2019).Usability Validation and the Probability of Use ErrorA specific area where sample size justification routinely fails regulatory scrutiny is in summative usability validation (IEC 62366-1:2015). The goal of usability validation is to demonstrate that the device can be used safely and effectively by the intended users without serious use errors.The FDA guidance on human factors engineering and the EU MDR requirements for usability engineering dictate that usability testing must sample a representative population of users. The statistical defence of this sample size estimate is uniquely challenging because use errors are ideally rare events.If a critical use error has an acceptable upper bound of 1 in 10,000 occurrences, demonstrating this via a standard c=0 attribute sampling plan would require an astronomical sample size:n = \frac{\ln(1 - 0.95)}{\ln(1 - 0.0001)} \approx 29,956Testing 30,000 participants in a usability study is generally impractical. Therefore, the statistical justification for usability testing cannot rely on frequentist attribute sampling alone. Under FDA human-factors guidance and IEC 62366-1, summative usability validation is treated as a primarily qualitative exercise. This entails surfacing use errors and analysing their root causes with roughly15 participants per user group serving as a problem-discovery heuristic, rather than a statistically powered estimate of an error rate. To strengthen the regulatory submission a biostatistician may supplement this qualitative validation with a Bayesian hierarchical model that combines the summative test data with prior data from formative studies, heuristic evaluations, and predicate device post-market surveillance. By establishing a prior distribution for the use error rate, the posterior distribution can be evaluated to provide additional quantitative assurance that the upper 95% credible bound for the critical use error rate falls below the acceptable threshold.This approach aligns with the MDR’s emphasis on continuous benefit-risk evaluation and the integration of post-market data into the clinical evaluation cycle.Documenting the Defence of a Sample Size EstimationA robust sample size justification section in a V&V protocol or CER must contain the following elements to survive Notified Body scrutiny:Objective Statement: Explicitly state whether the objective is to estimate a parameter (requiring a CI), bound a future observation (requiring a prediction interval), or bound a proportion of the population (requiring a tolerance interval).Statistical Framework: Specify the exact formula or distribution used. If using a c=0 plan, provide the binomial derivation. If using a Cpk plan, specify the non-central t-distribution parameters.Assumptions: Detail the assumed variance, the assumed defective rate, or the prior distribution. Provide the rationale for these assumptions based on pilot data or engineering analysis.Risk Integration: Map the statistical parameters (e.g., \alpha, \beta, p_0) directly to the risk acceptance criteria in the Risk Management File. If a hazard is classified as “Catastrophic,” the statistical assurance must be correspondingly high (e.g., 99% confidence).Sensitivity Analysis: If Bayesian assurance is used, demonstrate the impact of varying the prior distribution. If frequentist power is used, demonstrate the impact of varying the assumed effect size.ConclusionSample size justification in a medical device context is a multi-faceted statistical consideration that encompasses engineering constraints, clinical risk, and regulatory expectations. By use of acceptance sampling, the derivation of non-central t-distributions for Cpk, and the Bayesian integration of prior data via assurance calculations, a sample size justification can become a quantitative asset, rather than a regulatory vulnerability.References:Pardo, S. A. (2023). Statistical Methods and Analyses for Medical Devices. Springer.European Parliament and Council. (2017). Regulation (EU) 2017/745 on medical devices (MDR), Articles 61, 62, 83, and Annex XV. Official Journal of the European Union.International Organization for Standardization. (1999). ISO 2859-1:1999 – Sampling procedures for inspection by attributes.Squeglia, N. L. (2023). Zero Acceptance Number Sampling Plans (6th ed.). Quality Press.International Organization for Standardization. (2014). ISO 16269-6:2014 – Statistical interpretation of data – Part 6: Determination of statistical tolerance intervals.International Electrotechnical Commission. (2015). IEC 62366-1:2015 – Medical devices – Application of usability engineering to medical devices.US Food and Drug Administration. (2016). Applying Human Factors and Usability Engineering to Medical Devices.US Food and Drug Administration. (2010). Guidance for the Use of Bayesian Statistics in Medical Device Clinical Trials.

Series: Advanced Biostatistics for MedTech: Bridging Clinical Evaluation and EngineeringThe Regulatory Trap of “Substantial Equivalence”Regulatory pathways for medical devices rely heavily on the concept of equivalence. Under the US FDA 510(k) framework, a manufacturer must demonstrate that a new device is “substantially equivalent” to a legally marketed predicate. In Europe, the EU Medical Device Regulation (MDR 2017/745) Article 61 requires clinical evaluation data to demonstrate equivalent safety and performance when relying on predicate device data. As the MDR legacy-device transition deadlines approach (31 December 2027 for higher-risk devices (Class III and Class IIb implantable) and 31 December 2028 for lower-risk classes under Regulation (EU) 2023/607) Notified Bodies are applying unprecedented scrutiny to these equivalence claims. Equivalence demonstrations effectively require a comparable statistical standards as those required for pivotal clinical investigations.That said, the statistical rigour required for a particular device submission is proportional to the its risk classification and the degree of reliance on predicate data. Class I or low-risk IIa devices with a well-established predicate may suffice with descriptive comparisons and basic confidence intervals. Class IIb, III, and implantable devices, or any submissions introducing novel technological changes, increasingly demand the formal TOST or non-inferiority frameworks outlined below.The statistical theory of equivalence testing is well-established and, as such, its practical application in MedTech collides with legacy industry practices. Historically, substantial equivalence was often asserted retrospectively by leveraging a non-significant p-value from a conventional two-sided superiority test:H_0: \mu_1 = \mu_2 \quad \text{vs.} \quad H_1: \mu_1 \neq \mu_2Failing to reject H_0 merely indicates insufficient power to detect a difference, not proof of sameness. The institutional inertia to treat non-significant results as equivalence remains a persistent hurdle in device validation. If a clinical evaluation report (CER) submitted to a Notified Body relies on this absence-of-evidence fallacy, it constitutes a fatal statistical flaw that will trigger a Major Non-Conformity.To satisfy regulatory expectations under FDA guidance and MDR requirements, the biostatistician must enforce a paradigm shift in hypothesis formulation. This requires mandating formal equivalence and non-inferiority testing frameworks, including the pre-specification of clinically justified margins and the application of specialised power analyses.The Two One-Sided Tests (TOST) FrameworkThe statistical solution to demonstrating equivalence is the Two One-Sided Tests (TOST) procedure, originally formalised by Schuirmann (1987). Instead of testing for a difference, the TOST framework tests whether the true difference falls within a pre-defined equivalence interval bounded by -\Delta and +\Delta.The hypotheses are structured as:H_0: |\mu_1 - \mu_2| > \Delta \quad \text{vs.} \quad H_1: |\mu_1 - \mu_2| \leq \DeltaWhere \Delta is the equivalence margin. To test this, the procedure decomposes the hypothesis into two separate one-sided tests:Test 1 (Lower Bound):H_{0a}: \mu_1 - \mu_2 \leq -\Delta \quad \text{vs.} \quad H_{1a}: \mu_1 - \mu_2 > -\DeltaTest 2 (Upper Bound):H_{0b}: \mu_1 - \mu_2 \geq +\Delta \quad \text{vs.} \quad H_{1b}: \mu_1 - \mu_2 < +\DeltaBoth null hypotheses must be rejected to conclude equivalence. A critical property of the TOST procedure is its handling of the Type I error rate (\alpha). The TOST is an application of the Intersection-Union Test (IUT) principle. In an IUT, the overall null hypothesis is the union of individual null hypotheses, and the alternative is the intersection. The overall Type I error rate is bounded by the maximum of the individual test error rates and thus no multiplicity adjustment (such as Bonferroni) is required. Each one-sided test is evaluated at the 100(1-\alpha)th percentile, not the 100(1-\alpha/2)th percentile used in conventional two-sided testing.This amounts to constructing a 100(1-2\alpha)\% two-sided confidence interval around the observed mean difference and verifying that the entire interval falls within the equivalence bounds [-\Delta, +\Delta]. For a standard 95% confidence level (implying \alpha = 0.05), a 90% two-sided confidence interval is calculated and compared against the margins.The Justification of the Equivalence Margin (\Delta)The most contentious aspect of equivalence testing in medical device submissions is the justification of the margin, \Delta. In pharmaceutical bioequivalence studies, the margins (80% to 125% for pharmacokinetic parameters) are codified in regulation. No such codified margins exist for medical devices.As Pardo (2023) emphasises, the limits of equivalence or non-inferiority for medical devices are determined based on functionality and clinical context, not by arbitrary statistical thresholds. An engineer cannot set \Delta based on manufacturing tolerances, nor can a statistician set it based on standard deviations. The margin must represent the largest difference that is clinically acceptable within the specific context, such that a difference larger than \Delta would impact patient safety or clinical performance.Margin Derivation StrategiesUnder MDR, clinical equivalence must be demonstrated across clinical, biological, and technical parameters. When defending a CER or 510(k) submission, the biostatistician must document the derivation of \Delta using a multi-faceted approach:Clinical Significance Thresholds: \Delta may be anchored to the smallest change in a clinical endpoint that a patient or clinician would consider meaningful.State of the Art (SOTA) Benchmarking: Under MDR, \Delta can be derived from the observed performance variance of the SOTA. If the SOTA devices on the market exhibit a standard deviation (σ_SOTA) in a specific metric, a clinically justified fraction of this observed variability may be proposed as the equivalence margin. For instance, a manufacturer might argue that a difference not exceeding, say, 25% of the SOTA’s inherent variability is clinically negligible. This fraction must be explicitly anchored to patient-relevant outcomes and clinical expert consensus.Analytical Performance Goals: For In Vitro Diagnostics, \Delta is often tied to Total Analytical Error (TAE) specifications. If a clinical laboratory allows a maximum TAE of 10%, the equivalence margin for bias in a method comparison study might be set at \pm 5\% to leave adequate room for imprecision.If the margin is set too wide, the Notified Body will reject the claim as clinically meaningless. If set too narrow, the required sample size will become prohibitive. Margin derivation must be clinically justified and cannot be based solely on statistical convenience. For example, in orthopaedic implants, a difference in Range of Motion (ROM) of less than 5 degrees might be deemed clinically irrelevant, establishing Δ=5. Alternatively, Δ may be anchored to clinical significance thresholds or the observed performance variance of the SOTA. Any fraction of SOTA variance proposed must be explicitly justified in the Statistical Analysis Plan (SAP) with clinical and engineering rationale.The Danger of BiocreepA specific regulatory concern in equivalence testing is “biocreep.” If a manufacturer selects a slightly inferior predicate and sets a wide equivalence margin, the new device may be statistically equivalent to the predicate but clinically inferior to the original standard of care. Over successive iterations of 510(k) submissions, the performance of devices on the market can progressively degrade. Notified Bodies and the FDA are acutely aware of biocreep. To defend against this accusation, the SAP must demonstrate that the chosen margin \Delta is clinically relevant in the context of the SOTA, not merely in the context of the chosen predicate.Power and Sample Size for Equivalence TestingPower calculations for equivalence tests diverge significantly from superiority tests. In a superiority trial, power is at a minimum when the true difference is zero, and increases as the true difference moves away from zero. In an equivalence trial, power is maximised when the true difference is zero. A the true difference approaches the equivalence margin \Delta, the power drops precipitously toward the Type I error rate (\alpha).To illustrate: Δ = 5°, n = 100 per arm, SD = 10°, α = 0.05True differenceSuperiority powerEquivalence power0°0.0500.9401°0.1080.8752°0.2910.6803°0.5600.4075° (= Δ)0.9400.0507°0.9980.001The sample size for a two-group equivalence test (assuming equal variances and equal sample sizes) is derived using the non-central t-distribution. The non-centrality parameter (\delta_{ncp}) is a function of the assumed true difference (\Delta_a), the margin (\Delta_0), the standard deviation (\sigma), and the sample size per group (n):\delta_{ncp} = \frac{|\Delta_a - \Delta_0|}{\sigma \sqrt{2/n}}Assuming the true difference is zero (\Delta_a = 0), the formula simplifies. Regulatory bodies frequently require sponsors to power the study assuming a conservative non-zero true difference (e.g., assuming the true difference is 20% of the margin). The power (1 - \beta) is the probability that the test statistic exceeds the critical value under the alternative hypothesis.A large-sample normal approximation for the sample size per group (assuming the true difference is zero, \Delta_a = 0) is: n \approx \frac{2\sigma^2 (z_{1-\alpha} + z_{1-\beta/2})^2}{\Delta_0^2} Where z_{1-\alpha} and z_{1-\beta/2} are the standard normal quantiles. Note that z_{1-\beta/2} is used because power is distributed across both tails of the rejection region when the true difference is zero. This is an approximate formula. Exact calculations using the t‑distribution (as implemented in PowerTOST) will give slightly larger required n.A critical error in submission preparation is using standard superiority sample size calculators for equivalence designs. Doing so routinely underpowers the study which leads to failed verification and validation (V&V) activities.The TOST procedure can be inherently conservative. Test statistic relies on the non-central t-distribution which means the actual Type I error rate is \leq \alpha. At the boundary the test statistic follows a central t‑distribution (non‑centrality = 0), giving exactly α. Away from the boundary it is conservative. While statistically safe, it means that standard sample size formulas (which rely on normal approximations) may slightly underestimate the required sample size. Exact power calculations using purpose-built equivalence tools are critical for regulatory submissions. These can be achieved with the PowerTOST or TOSTER packages in R (which implement the non-central t solution for TOST), or dedicated software such as PASS or SAS PROC POWER. General superiority-power functions such as base R’s power.prop.test or the pwr package’s pwr.t.test are not designed for TOST and will misstate the required sample size.Non-Inferiority: The One-Sided AlternativeIn many medical device regulatory applications, particularly those involving design modifications intended to reduce cost, improve usability, or minimise invasiveness, the goal is not to prove exact equivalence, but to prove that the new device is “no worse” than the predicate. This is the domain of non-inferiority testing.Non-inferiority tests are one-sided. The hypotheses are:H_0: \mu_1 - \mu_2 \leq -\Delta \quad \text{vs.} \quad H_1: \mu_1 - \mu_2 > -\DeltaHere, \Delta is the non-inferiority margin. If the outcome is such that higher values are better (e.g., tensile strength, sensitivity), the new device is non-inferior if its mean is not more than \Delta units below the predicate’s mean. The statistical defence of this claim requires constructing a 100(1-\alpha)\% one-sided confidence interval and verifying that the lower bound does not cross -\Delta.Non-inferiority testing is particularly relevant under MDR for justifying the use of historical data. If a manufacturer modifies an existing device (e.g., changing a polymer supplier), the clinical evaluation can leverage the historical safety data of the predicate, provided a non-inferiority test on the critical quality attributes (CQAs) is statistically valid.The Constancy AssumptionA key statistical requirement for non-inferiority is the constancy assumption, borrowed from pharmaceutical context (ICH E10). This assumption requires that the historical predicate’s performance, established in previous studies, would remain unchanged if those studies were conducted today. In medical device development, where surgical techniques, imaging technologies, and patient management protocols evolve rapidly, the constancy assumption is often violated. If the standard of care has improved since the predicate was launched, a non-inferiority margin based on the predicate’s historical performance may be clinically obsolete. A biostatistician must assess whether the historical data supporting the margin is still valid in the current clinical context.Equivalence and Non-Inferiority for Binary OutcomesWhile continuous outcomes are relatively straightforward, binary outcomes (e.g., pass/fail rates, clinical success rates, sensitivity) present a more complex challenge. The sampling distributions of differences in proportions (p_1 - p_2) and odds ratios (OR) are more difficult to approximate reliably in small samples, see Pardo (2023).For binary outcomes, the TOST framework is applied to the difference in proportions. The equivalence hypotheses are:H_0: |p_1 - p_2| > \Delta \quad \text{vs.} \quad H_1: |p_1 - p_2| \leq \DeltaTo compute confidence intervals for the difference in proportions or the odds ratio, standard normal approximations (e.g., the Wald interval) exhibit poor coverage probability when sample sizes are small or when proportions are near 0 or 1. While methods like the Farrington-Manning score test or exact unconditional tests are standard regulatory choices, Pardo notes the lack of a simple closed-form sampling distribution for these metrics and instead discusses bootstrap resampling as a robust, assumption-light alternative.Bootstrap Resampling for Equivalence IntervalsThe bootstrap procedure for an equivalence test of an odds ratio involves:Computing the observed sample proportions \hat{p}_1 and \hat{p}_2, and the observed odds ratio \widehat{OR}.For N repetitions (e.g., N = 10,000), drawing a random sample of size n_1 with replacement from group 1, and a random sample of size n_2 with replacement from group 2.Calculating the resampled odds ratio OR^*_i for each iteration i.Determining the 100(\alpha)th and 100(1-\alpha)th percentiles of the resulting OR^* distribution to form the two-sided 100(1-2\alpha)\% confidence interval.Verifying that this percentile interval falls entirely within the pre-specified equivalence bounds for the odds ratio.Using the percentile bootstrap method provides a robust, assumption-light justification of equivalence when submitting binary outcome data to regulatory reviewers.Variance Non-Inferiority: The Statistical Defence of ImprecisionIn medical device validation there is a regulatory requirement to control imprecision. Where a new device’s repeatability, reliability, or overall dispersion is a critical determinant of safety or clinical performance, Notified Bodies and the FDA require a formal statistical justification of variance to ensure the new device is not unacceptably more variable than the comparator. The frequency of this requirement depends on the device type and the specific performance characteristics under evaluation.Standard F-tests are used to compare the variances of two independent samples. In device validation, however, the objective is often to demonstrate that the evaluation device’s standard deviation (\sigma_e) is “no worse” than the comparator’s (\sigma_c) by more than a pre-specified factor k_0. This is a non-inferiority test on standard deviations.The hypotheses are:H_0: \sigma_e > k_0 \sigma_c \quad \text{vs.} \quad H_1: \sigma_e \leq k_0 \sigma_cWhere k_0 \geq 1. If k_0 = 1.2, the assertion is that the new device’s standard deviation is no more than 20% larger than the predicate’s.The test statistic utilises the F-distribution. Under the null hypothesis, the ratio of the evaluation variance to the scaled comparator variance is distributed as F with (n_e - 1) and (n_c - 1) degrees of freedom:F = \frac{S_e^2}{k_0^2 S_c^2}Where S_e and S_c are the sample standard deviations. The null hypothesis is rejected if F < F_{\alpha, n_e-1, n_c-1}, where F_{\alpha} is the \alpha-quantile of the F-distribution.The power of this test depends on the true ratio of variances. Let the true ratio of standard deviations be k_1 = \sigma_e / \sigma_c. We define the variance ratio parameter as \delta_V = (k_1 / k_0)^2. As \delta_V decreases (i.e., the evaluation device is actually more precise than the comparator), the probability of rejecting the null hypothesis increases. As Pardo (2023) details, the power curve for this non-inferiority test is asymmetric, and sample size justification must be derived iteratively. The power is formally calculated as the probability that the test statistic falls below the critical value:P \left( F_{n_e-1, n_c-1} < \frac{F_{\alpha, n_e-1, n_c-1}}{\delta_V} \right) , where δ_V = (k₁ / k₀)²This is typically evaluated via simulation or the F‑distribution with degrees of freedom (n_e-1, n_c-1) and scale factor δ_V to provide adequate power at the assumed true variance ratio.Documenting the Statistical Defence for RegulatorsTo survive a Notified Body audit or an FDA 510(k) review, the statistical justification of equivalence or non-inferiority must be documented exhaustively. A sound submission includes:Pre-Specification: The SAP must specify the TOST or non-inferiority framework, the margin (\Delta or k_0), the \alpha level, and the power calculation prior to data collection. Post-hoc margin adjustment is strictly prohibited.Margin Justification: A dedicated section of the CER must justify \Delta using clinical, engineering, and SOTA data. The justification must explicitly state why a difference larger than \Delta would be unsafe or clinically unacceptable.Assumption Verification: The submission must include diagnostic plots verifying the assumptions of the test (e.g., normality of residuals, homoscedasticity). If assumptions are violated, the submission must detail the use of robust alternatives (e.g., Passing-Bablok, bootstrap).Intent-to-Treat vs. Per-Protocol: For clinical investigations, equivalence analyses are typically evaluated on both Per-Protocol (PP) and Intent-to-Treat (ITT) populations as co-primary analyses. In superiority trials, ITT is the conservative choice because protocol deviations tend to dilute the treatment effect. In equivalence and non-inferiority trials, protocol deviations can artificially narrow the difference between devices, making ITT anti-conservative. Conversely, PP can be biased if dropouts are related to treatment failure. Current FDA and EMA guidance therefore expects both ITT and PP to be analysed and the reasons for the differences in population, if any, must be explored and documented.SummaryTo demonstrate substantial equivalence or non-inferiority is a regulatory cornerstone of the medical device industry. It cannot be achieved through the misapplication of standard superiority testing. Application of the TOST framework is essential for regulatory success, combined with clinically justified margins and appropriate power analyses using the non-central t-distribution. For binary outcomes and complex variance comparisons, bootstrap resampling and non-central F-distributions provide the robust statistical defence required to satisfy Notified Body expectations.References:Pardo, S. A. (2023). Statistical Methods and Analyses for Medical Devices. Springer.European Parliament and Council. (2017). Regulation (EU) 2017/745 on medical devices (MDR), Article 61 and Annex XIV. Official Journal of the European Union.US Food and Drug Administration. (2019). The 510(k) Program: Evaluating Substantial Equivalence in Premarket Notifications [510(k)].Schuirmann, D. J. (1987). A comparison of the two one-sided tests procedure and the power approach for assessing the equivalence of average bioavailability. Journal of Pharmacokinetics and Biopharmaceutics, 15(6), 657-680.International Council for Harmonisation. (2000). ICH E10: Choice of Control Group and Related Issues in Clinical Trials.US Food and Drug Administration. (2016). Non-Inferiority Clinical Trials to Establish Effectiveness: Guidance for Industry.European Medicines Agency. (2005). Guideline on the Choice of the Non-Inferiority Margin.

Series: Advanced Biostatistics for MedTech: Bridging Clinical Evaluation and EngineeringThe Regulatory Demand for Analytical Rigour in IVDsThe transition from the In Vitro Diagnostic Directive (IVDD 98/79/EC) to the In Vitro Diagnostic Regulation (IVDR 2017/746) has fundamentally altered the analytical performance expectations for medical devices in the EU/EEA (and Northern Ireland under the Windsor Framework), with Great Britain accepting IVDR-compliant CE-marked devices transitionally. Under IVDR Annex I (General Safety and Performance Requirements), manufacturers must demonstrably justify the analytical performance of their devices, including trueness (bias), precision (imprecision), accuracy, and measurement uncertainty.For In Vitro Diagnostics (IVDs) and continuous monitoring devices (such as blood glucose meters (BGMs), continuous glucose monitors (CGMs), and high-sensitivity immunoassays) demonstrating analytical performance necessitates a method comparison study against a reference measurement procedure or a validated comparator. A widely recognised consensus standard for designing and analysing these studies is the Clinical and Laboratory Standards Institute (CLSI) guideline EP09c, Measurement Procedure Comparison and Bias Estimation Using Patient Samples. This is standard recognised by regulators, including the FDA.Pardo (2023) introduces the foundational linear regression concepts for comparing an evaluation method to a comparator, initially presenting the simple case where the evaluation method has variability while the comparator method’s variability is assumed negligible. However, applying standard Ordinary Least Squares (OLS) regression to IVD method comparison data often violates this simplifying assumption. Relying on OLS in regulatory submissions for IVDs can lead to biased (attenuated) slope estimates and immediate technical deficiencies during Notified Body reviews. The correlation coefficient often cited in its support is itself a poor measure of agreement. It is inflated by the width of the concentration range, not by true concordance.This article dissects the advanced statistical methodologies required for IVD method comparison under IVDR, focusing on the transition from OLS to Deming and Passing-Bablok regression, the handling of heteroscedasticity, and the deployment of Bayesian hierarchical models for multi-site analytical validation.The Statistical Inadequacy of Ordinary Least Squares in Method ComparisonThe fundamental premise of a method comparison study is to evaluate the agreement between two measurement methods: the reference method (X) and the evaluation method (Y) across a range of analyte concentrations. The standard linear model is:Y = \beta_0 + \beta_1 X + \epsilonIn OLS regression, the minimisation objective is the sum of squared residuals, performed exclusively in the Y direction:\min \sum_{i=1}^{n} (Y_i - \hat{Y}_i)^2The critical assumption underlying OLS is that the independent variable (X) is measured without error, and all random error is contained within the dependent variable (Y). In IVD method comparison, both the reference method and the evaluation method possess measurement error. If X is measured with error, the observed values are X_{obs} = X_{true} + \delta, where \delta represents the random analytical error of the comparator.When OLS is applied to data where the independent variable contains error, it results in regression dilution (attenuation bias). The OLS estimator for the slope is biased toward zero:E[\hat{\beta}<em>{1,OLS}] = \beta_1 \left( \frac{\sigma</em>{true}^2}{\sigma_{true}^2 + \sigma_{\delta}^2} \right)Where \sigma_{true}^2 is the variance of the true analyte concentration and \sigma_{\delta}^2 is the variance of the comparator’s measurement error. In high-precision IVD systems where the analytical measurement range (AMR) is narrow relative to the measurement error, this attenuation bias becomes substantial. Submitting an OLS-derived slope and intercept to a Notified Body under IVDR is statistically indefensible, as it attenuates the slope toward zero, distorting the proportional-bias estimate in a direction that depends on the true slope, while systematically biasing the intercept upward.Deming Regression: Minimising Weighted DistancesTo address the error-in-variables problem, CLSI EP09c recommends regression methods that account for error in both variables; Deming regression is one of the principal approaches presented, alongside Passing-Bablok and other techniques, with the choice depending on the data characteristics and assumptions. Unlike OLS, Deming regression accounts for measurement error in both X and Y by minimising the weighted sum of squared residuals in both directions.The Deming minimisation objective is:\min \sum_{i=1}^{n} \left[ \frac{(Y_i - \hat{Y}<em>i)^2}{\sigma</em>{\epsilon}^2} + \frac{(X_i - \hat{X}<em>i)^2}{\sigma</em>{\delta}^2} \right]Where \sigma_{\epsilon}^2 is the variance of the evaluation method’s measurement error, and \sigma_{\delta}^2 is the variance of the comparator method’s measurement error. This is often parameterised using the variance ratio \lambda = \sigma_{\epsilon}^2 / \sigma_{\delta}^2. The resulting slope estimator (\hat{\beta}_{1,Deming}) is consistent, provided that the variance ratio \lambda is correctly specified (or, in large samples, consistently estimated).The estimation of \lambda requires a separate precision validation study, typically conducted in accordance with CLSI EP05 (Evaluation of Precision of Quantitative Measurement Procedures). If the variance ratio is assumed to be 1 (i.e., the error variances of both methods are equal), the technique is referred to as orthogonal regression. Assuming \lambda = 1 without empirical justification is a frequent pitfall in regulatory submissions. For IVDs, the reference method (e.g., isotope dilution mass spectrometry) typically exhibits significantly lower variance than the evaluation method (e.g., an enzymatic point-of-care test). A defensible IVDR submission must derive \lambda from replicates measured across the AMR.Passing-Bablok Regression: Robustness Against Non-Normality and OutliersWhile Deming regression resolves the error-in-variables bias, it remains a parametric technique dependent on the assumption that the measurement errors are normally distributed and free of gross outliers. Clinical samples frequently contain outliers due to haemolysis, lipaemia, or interfering substances.Passing-Bablok regression is a non-parametric alternative recommended by CLSI EP09c when the data distribution is non-normal or when outliers are present. The method estimates the slope (\beta_1) by taking the median of all possible pairwise slopes of the lines connecting points in the dataset, adjusted to be robust to the exchange of X and Y axes.For n data points, there are N = n(n-1)/2 pairwise slopes S_{ij}:S_{ij} = \frac{Y_j - Y_i}{X_j - X_i} \quad \text{for } i < jThe Passing-Bablok slope estimator is:\hat{\beta}<em>1 = \operatorname{median} S</em>{ij}Specifically, it is the median of the shifted pairwise slopes, with a correction for those slopes where the sign flips if X and Y are swapped. The intercept is subsequently derived as:\hat{\beta}_0 = \operatorname{median} (Y_i - \hat{\beta}_1 X_i)The primary advantage of Passing-Bablok in the medtech regulatory context is its robustness. A single outlier that would severely distort an OLS or even a Deming regression estimate has effectively negligible influence on the Passing-Bablok median. The estimator tolerates a substantial minority of outliers, providing robustness where standard parametric methods fail. Confidence intervals for the slope and intercept are constructed using the method’s own analytic, rank-based procedure, derived by Passing and Bablok from the distribution of the ranked pairwise slopes. Bootstrap resampling is an available alternative to this when the analytic assumptions are questionable.The trade-off is statistical power. If the errors are truly normal and homoscedastic, Passing-Bablok is less efficient than Deming regression. The choice between Deming and Passing-Bablok must be governed by a decision rule fixed in the SAP, with the triggering diagnostics and their thresholds stated in advance of any analysis of study data. For example: Deming regression will be used as the primary method, with Passing-Bablok substituted if the Shapiro-Wilk test on the residuals of a preliminary fit returns p < 0.05, or if more than a stated proportion of observations exceed a pre-specified Cook’s distance threshold. Specifying the rule, the triggering diagnostics and the thresholds in advance keeps the choice pre-specified while allowing it to respond to the data’s actual distributional behaviour.Handling Heteroscedasticity: The Achilles Heel of Bland-AltmanBland-Altman analysis (difference plots) is a staple of method comparison, focusing on the agreement between two methods rather than the mathematical relationship. The method plots the difference between methods (Y_i - X_i) against their average (\frac{Y_i + X_i}{2}).The fundamental limitation of standard Bland-Altman analysis is its assumption of homoscedasticity, that the variance of the differences is constant across the entire measurement range. For the vast majority of IVDs, particularly continuous biosensors and immunoassays, this assumption is violently violated. Analytical imprecision typically scales with analyte concentration.Consider a continuous glucose monitor evaluated against a laboratory hexokinase reference method. At hypoglycaemic concentrations (e.g., 2.0 \text{ mmol/L}), the absolute difference between methods may be \pm 0.3 \text{ mmol/L}. At hyperglycaemic concentrations (e.g., 20.0 \text{ mmol/L}), the absolute difference may be \pm 3.0 \text{ mmol/L}. If standard Bland-Altman limits of agreement are applied, the limits will be excessively wide at the low end (overestimating clinically acceptable error) and excessively narrow at the high end (masking clinically significant deviation).Variance-Stabilising TransformationsThe traditional regulatory approach to heteroscedastic method comparison data is to apply a variance-stabilising transformation, such as the logarithmic transformation. If the coefficient of variation (CV) is constant, the variance of the differences scales with the square of the mean. A log transformation converts proportional error into additive error.Logarithmic transformations complicate clinical interpretation.A Notified Body reviewer assessing a BGM against ISO 15197:2013 criteria needs to evaluate error in clinically meaningful units (mg/dL or mmol/L), not in log-transformed space. Back-transforming Bland-Altman limits from log space yields geometric means and ratios, which are frequently misinterpreted by clinical reviewers.Weighted Deming Regression and Percentage ErrorFor heteroscedastic data where transformation is undesirable, Weighted Deming Regression is the mathematically rigorous solution provided a reliable variance function is available. Instead of a constant variance ratio \lambda, the weights are allowed to vary across the concentration range. The variance of the measurement error is modelled as a function of the analyte concentration.A common model for the variance function in IVDs is a power function:\sigma^2 (X) = a + b X^cWhere c determines the relationship between concentration and variance. By estimating this function (often via replicates at specific concentration levels), the Deming regression can be weighted inversely proportional to the variance at each point. This ensures that high-concentration points with inherently larger absolute variance do not unduly influence the estimation of the slope and intercept.Both the variance ratio \lambda and the variance function \sigma^2(X) require replicate measurements distributed across the AMR. This is a design commitment rather than an analysis choice: the precision study (CLSI EP05) and the method comparison must be planned together, with the number of concentration levels and the replicates per level specified in the protocol. A study designed around single measurements of patient samples cannot support weighted Deming retrospectively.Regulatory standards like ISO 15197 for blood glucose monitoring systems explicitly define acceptable error as a combination of absolute and relative criteria (e.g., \pm 15 \text{ mg/dL} for glucose < 100 \text{ mg/dL}, and \pm 15\% for glucose \ge 100 \text{ mg/dL}). The statistical evaluation of method comparison data against such criteria requires the calculation of Parkes Error Grids or Surveillance Error Grids, which map the clinical risk of the observed differences. The biostatistician must integrate the outputs of the method comparison regression (bias and imprecision estimates) into these clinical risk frameworks to satisfy IVDR requirements for analytical performance.The Total Analytical Error (TAE) FrameworkUnder IVDR Annex I, Section 9.1(a), manufacturers must specify and justify analytical performance characteristics – including trueness (bias), precision (imprecision), and accuracy. Separately, Section 9.3 establishes metrological traceability requirements, which lean toward the ISO/GUM measurement-uncertainty approach. While the regulation does not prescribe a single consolidated metric, in clinical-chemistry practice, these components are frequently combined into the Total Analytical Error (TAE) framework (formalised in CLSI EP21), which reflects the maximum expected error of a single measurement. While CLSI EP21 does not prescribe this as the only calculation method, a common approximation used in clinical chemistry is the one-sided Westgard model. It is worth noting that TAE coexists with, and is often contrasted against, the measurement-uncertainty approach.The mathematical formulation of this TAE approximation is:TAE = |\text{Bias}| + 1.65 \times SDOr, utilising the coefficient of variation:TAE\% = |\text{Bias}\%| + 1.65 \times CV\%The factor 1.65 corresponds to the 95th percentile of a one-sided normal distribution. This reflects the regulatory expectation that 95% of all individual measurements will fall within this bound above (or below) the target value; a symmetric two-sided bound is a different (and separately debated) construction.While the formula is simple, its application in method comparison is nuanced. The bias is not a single number; it is derived from the method comparison regression. If Deming regression yields a slope \hat{\beta}_1 = 1.05 and an intercept \hat{\beta}_0 = 2.0 \text{ mg/dL}, the bias is concentration-dependent:\text{Bias}(X) = \beta_0 + (\beta_1 - 1)XSimilarly, the imprecision (SD or CV) is derived from the precision study (CLSI EP05) and is typically heteroscedastic. Therefore, the TAE must be calculated and plotted across the entire AMR. A Notified Body will reject a submission that presents a single TAE value if the device exhibits proportional bias or concentration-dependent imprecision. The biostatistician must generate a TAE plot, overlaying the calculated TAE against the manufacturer’s claimed performance specifications, demonstrating that the upper bound of TAE does not exceed the allowable total error (ATE) at any point in the clinical reportable range.Advanced Application: Bayesian Hierarchical Models for Multi-Site ValidationWhile IVDR does not explicitly mandate a specific number of testing sites, analytical performance is expected to capture inter-laboratory variability. The convention of using three distinct clinical laboratories stems from general regulatory practice and Notified Body expectations, while device‑specific standards such as ISO 15197 add concrete requirements (three reagent system lots and three sites) without altering the higher‑level IVDR expectation that inter‑laboratory variability be captured. The traditional frequentist approach involves calculating TAE at each site independently and either pooling the data or taking the worst-case scenario.Pooling the data assumes that inter-site variability is negligible. If inter-site variability is significant (e.g., due to differences in reagent handling, altitude, or operator training), pooling biases the variance estimates. Taking the worst-case site is overly conservative and reduces the statistical power of the validation.A statistically robust approach is the application of Bayesian Hierarchical Models (also known as multilevel models). This framework allows for the simultaneous estimation of site-specific performance parameters while borrowing strength across sites to stabilise estimates. While not mandated by IVDR or CLSI guidance, this approach offers a sophisticated treatment of multi-site variability.Consider a method comparison study conducted across J sites, with n_j samples per site. The hierarchical model for the method comparison relationship can be formulated as:Level 1 (Within-Site):Y_{ij} = \alpha_j + \beta_j X_{ij} + \epsilon_{ij}Level 2 (Between-Site):\alpha_j \sim N(\mu_\alpha, \tau_\alpha^2)\beta_j \sim N(\mu_\beta, \tau_\beta^2)Where:Y_{ij} is the evaluation method result for sample i at site j.X_{ij} is the comparator method result.\alpha_j and \beta_j are the site-specific intercept and slope.\mu_\alpha and \mu_\beta are the global (population-level) intercept and slope.\tau_\alpha^2 and \tau_\beta^2 represent the variance of the intercepts and slopes across sites.While this independent univariate normal formulation is the standard teaching model, site slope and intercept are usually correlated. A fully specified model therefore employs a bivariate normal distribution with a covariance term to account for this joint relationship.Using Markov Chain Monte Carlo (MCMC) methods, as described by Pardo (2023) in the context of Bayesian inference, we can sample from the posterior distributions of these parameters.The advantage of this approach under IVDR is twofold. First, it provides a formal statistical test for inter-site variability. If the posterior distributions of \tau_\alpha^2 and \tau_\beta^2 are tightly clustered around zero, the manufacturer can statistically justify pooling the data. If they are significantly greater than zero, the hierarchical model naturally accounts for this variability without resorting to worst-case arithmetic. Second, the hierarchical model yields a posterior distribution for the global TAE, incorporating both the within-site analytical uncertainty and the between-site variance. This provides a far more robust and defensible estimate of analytical performance than traditional frequentist aggregation.ConclusionMethod comparison for IVDs under IVDR is not a matter of plotting Y against X and fitting a trendline in Excel. The regulatory expectations demand rigorous treatment of measurement error, heteroscedasticity, and multi-site variability. The ability to defend a Deming regression slope or a hierarchical TAE estimate before a Notified Body reviewer is what differentiates a regulatory asset from a procedural bottleneck.The selection of Deming versus Passing-Bablok regression must be statistically justified based on error distributions and outlier diagnostics. Heteroscedasticity must be addressed, either through variance-stabilising transformations or weighted regression techniques, to ensure that Bland-Altman limits of agreement are clinically meaningful across the analytical measurement range. The integration of method comparison data into the Total Analytical Error framework and its extension via Bayesian hierarchical modelling for multi-site studies represents current best practices.References:Bland, J. M., & Altman, D. G. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet, *327*(8476), 307-310.Clinical and Laboratory Standards Institute. (2014). EP05-A3: Evaluation of precision of quantitative measurement procedures (3rd ed.). CLSI.Clinical and Laboratory Standards Institute. (2016). EP21: Evaluation of total analytical error for quantitative medical laboratory measurement procedures (2nd ed.). CLSI.Clinical and Laboratory Standards Institute. (2018). EP09c: Measurement procedure comparison and bias estimation using patient samples (3rd ed.). CLSI.European Parliament and Council. (2017). Regulation (EU) 2017/746 on in vitro diagnostic medical devices (IVDR). Official Journal of the European Union.Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2013). Bayesian data analysis (3rd ed.). Chapman & Hall/CRC.International Organization for Standardization. (2013). ISO 15197:2013 – In vitro diagnostic test systems – Requirements for blood-glucose monitoring systems for self-testing in managing diabetes mellitus. ISO.Linnet, K. (1993). Evaluation of regression procedures for methods comparison studies. Clinical Chemistry, *39*(3), 424-432.Pardo, S. A. (2023). Statistical methods and analyses for medical devices. Springer.Parkes, J. L., Slatin, S. L., Pardo, S., & Ginsberg, B. H. (2000). A new consensus error grid to evaluate the clinical significance of inaccuracies in the measurement of blood glucose. Diabetes Care, *23*(8), 1143-1148.Passing, H., & Bablok, W. (1983). A new biometrical procedure for testing the equality of measurements from two different analytical methods. Journal of Clinical Chemistry and Clinical Biochemistry, *21*(11), 709-720.

Series: Advanced Biostatistics for MedTech: Bridging Clinical Evaluation and EngineeringThe Pharma Paradigm vs. The MedTech RealityGraduate-level biostatistics curricula are largely rooted in the pharmaceutical paradigm. Academic training typically focuses on ICH E9 statistical principles, large-scale randomised controlled trials (RCTs), adaptive dose-finding, and survival analysis using Cox proportional hazards models. The statistical machinery is built to prove the efficacy of a chemical molecule against a placebo or standard of care, operating under the assumption that the intervention is chemically identical across batches and remains static throughout its lifecycle.The medical device industry operates under fundamentally different statistical mechanics. A pacemaker lead, a continuous glucose monitor (CGM), an in vitro diagnostic (IVD) reagent, or an AI-driven Software as a Medical Device (SaMD) does not exist in a vacuum. These products interact dynamically with biological tissue, exhibit inherent manufacturing variability, degrade over time, and are frequently subject to iterative design changes post-market launch.Under the EU Medical Device Regulation (MDR 2017/745), the In Vitro Diagnostic Regulation (IVDR 2017/746), and the UK MHRA framework (UKCA marking, alongside continued recognition of CE-marked devices), the regulatory emphasis shifts dramatically. Conformity assessment relies heavily on Clinical Evaluation Reports (CERs), State of the Art (SOTA) benchmarking, substantial equivalence demonstrations, and thorough post-market surveillance (PMS).As noted in Statistical Methods and Analyses for Medical Devices (Pardo, 2023), there are no statistical methods designed specifically for the analysis of medical device data. Methods such as Accelerated Life Testing (ALT) for shelf-life estimation, Receiver Operating Characteristic (ROC) curve assessment for diagnostics, acceptance sampling for lot release, and variance components analysis for manufacturing validation appear with regularity in device development and regulatory submissions. The challenge lies not in the statistical novelty of these methods, but in their correct application within a hybrid engineering-clinical regulatory framework that standard biostatistics programmes do not address.The Regulatory Crucible: EU MDR, IVDR, and MEDDEV 2.7/1 rev 4The regulatory framework governing medical devices in Europe and the UK creates a set of statistical expectations that diverge sharply from those encountered in pharmaceutical development.EU MDR Article 61 and Annex XIV mandate that clinical evaluation be based on a continuous, proactive process. It is not a one-off submission event. The MEDDEV 2.7/1 rev 4 guideline – retained by convention with no formal MDR status, alongside the MDCG 2020-series (2020-5, 2020-6, 2020-13) – explicitly requires a detailed, statistically justified methodology for appraising clinical data. Notified Bodies (e.g., BSI, TÜV SÜD, DEKRA) employ clinical evaluators who are trained to scrutinise the statistical rationale underpinning every claim in a CER.When a Notified Body reviews a technical documentation file, the clinical and statistical reviewers typically interrogate four core areas:Sample size justification: What statistical framework was used to determine the number of units tested, subjects enrolled, or sites included? Was the justification based on power analysis, tolerance intervals, or acceptance sampling logic – and is the chosen framework appropriate for the question being asked?Equivalence demonstration: What is the statistical basis for the claim of substantial equivalence to a predicate device? Were formal equivalence tests (e.g., Two One-Sided Tests, or TOST) employed, or was equivalence inferred from a failure to reject a null hypothesis of no difference?SOTA benchmarking: How was the “State of the Art” defined quantitatively? Were the comparator data synthesised using meta-analytic techniques, and was heterogeneity between sources formally assessed?Post-market surveillance thresholds: In the PMS plan, what statistical rules trigger a corrective and preventive action (CAPA)? Are control limits based on historical baselines, and are they updated dynamically?While pharmaceutical development also employs complex equivalence designs (e.g., for biosimilars) and adaptive frameworks, standard clinical biostatistics training predominantly focuses on powering trials to detect a minimally clinically important difference (MCID) in a patient population. In medtech, sample size justification must frequently accommodate bench testing, biocompatibility, usability engineering (IEC 62366-1:2015), and clinical investigation, each governed by different statistical logic and documented in distinct reports. Bench testing relies on tolerance intervals and reliability confidence levels. Usability validation relies on the probability of use error. Clinical investigations may employ adaptive or Bayesian designs. The medtech biostatistician must navigate all of these paradigms, ensuring statistical consistency and alignment across the entire Technical Documentation File, even though the clinical and engineering data reside in separate documents.The “State of the Art” Statistical ChallengeUnder MDR, demonstrating conformity with General Safety and Performance Requirements (GSPRs) requires comparison against the SOTA. The SOTA is a moving target – it represents the current best-in-class alternative, not a placebo. From a statistical perspective, this means that device clinical investigations are rarely superiority trials. They are most commonly equivalence or non-inferiority trials.The margins ( \Delta ) in device equivalence testing are not based on clinical consensus from massive RCTs, as they often are in bioequivalence studies for generic drugs. Device equivalence margins are typically derived from engineering tolerances, analytical performance goals (e.g., CLSI EP09c criteria for IVDs), or historical performance data of predicate devices. This creates a tension between the engineering team, which may set margins based on what the manufacturing process can achieve, and the regulatory requirement that margins be clinically justified.The statistical consequence is severe. If equivalence margins are set too wide, the Notified Body will reject the claim as clinically meaningless. If they are set too narrow, the sample size required to demonstrate equivalence becomes prohibitive. The biostatistician’s role is to broker a defensible middle ground, documenting the statistical and clinical rationale for the margin in a way that satisfies both engineering and regulatory reviewers.The Hybrid Data EcosystemMedical devices generate data that at times defies the neat categorisation found in clinical data standards (such as CDISC), which are mandated for pharmaceutical regulatory submissions but not universally required for medical device files. A medical device generates a hybrid data ecosystem comprising:Engineering/Bench Data: Tensile strength, fatigue life, electromagnetic compatibility (EMC), fluid dynamics, and dimensional measurements.Analytical/Sensor Data: Biosensor accuracy, signal-to-noise ratios, assay precision, and calibration curve performance.Clinical Data: MACE (Major Adverse Cardiovascular Events), patient-reported outcomes (PROs), and clinician assessments.Real-World Data (RWD): Post-market telemetry, software usage logs, complaint data, and registry data.Pardo’s text is notable because it addresses this hybrid ecosystem directly. The text moves fluidly between ANOVA for factorial experiments in product design, control charts for manufacturing process monitoring, and Kaplan-Meier estimation for censored time-to-failure data – all within the context of a single device lifecycle. This fluidity is essential because statistical decisions made during the engineering phase propagate directly into the clinical evaluation and post-market phases.Consider the relationship between manufacturing variance and clinical risk. An implantable device – such as a hip replacement or a cardiac stent – has inherent manufacturing variability in its dimensions, material properties, and surface finish. Under EU MDR, the manufacturer must demonstrate that this variability does not adversely affect clinical performance. This requires partitioning the variance of a critical quality attribute (CQA) between different manufacturing lots, different operators, and the inherent device variability – a task that calls for Variance Components Analysis (VCA) using Restricted Maximum Likelihood (REML) estimation, implemented via mixed-effects models (e.g., lmer() in R).If a Notified Body queries whether a slight change in a manufacturing process is acceptable, the response cannot be limited to “the mean remained the same.” The response must statistically demonstrate that the variance component attributed to the new process does not inflate the total variance of the device’s clinical performance. REML variance estimates must be translated into risk probabilities, and those probabilities must be mapped back to the risk management file (ISO 14971:2019).Statistical Blind Spot 1: The Misinterpretation of Confidence in ValidationIn academic statistics, confidence intervals are taught as a fundamental inferential tool. The Neyman definition states that a 95% confidence interval is an interval constructed from empirical observations that has a 95% probability of containing the true, unknown population parameter. It is a property of the procedure, not of any particular interval.In medical device Verification & Validation (V&V), this academic definition is frequently misapplied. Pardo (2023) provides a solid dissection of this issue. Engineers and non-statistical reviewers often ask for the sample size that will “give 95% confidence to pass the test.” This question is statistically meaningless. As Pardo notes, 100% confidence is achievable with a sample size of n=0 – the interval is simply the entire range of possible values for the parameter. Confidence is a post-hoc property; it is constructed after data are gathered. Sample size is driven by power (the a priori probability of rejecting a false null hypothesis) or by the desired width of a tolerance interval, not by “confidence” alone.The more profound issue is that medtech validation frequently requires tolerance intervals, not confidence intervals. A 95% confidence interval characterises the location of a population parameter (e.g., the mean). A 95/95 tolerance interval states that there is 95% confidence that 95% of the individual units in the population fall within these limits. When validating a physical dimension against a specification limit, the Notified Body expects evidence that virtually all individual units will conform – not just that the average unit conforms.Under ISO 16269-6, tolerance intervals require calculations based on non-central t-distributions or chi-squared approximations. These are methods that appear in advanced statistical theory courses but are rarely applied in standard biostatistics training programmes focused on clinical trials. The distinction between confidence intervals (for parameters), prediction intervals (for a single future observation), and tolerance intervals (for a proportion of the population) is not merely academic; it is a frequent source of Major Non-Conformities during Notified Body audits.Statistical Blind Spot 2: Acceptance Sampling and the c=0 ParadigmIn pharmaceutical manufacturing, every tablet in a batch is assumed to be chemically identical within tight dissolution and content uniformity specifications. In medical device manufacturing, particularly for high-volume products such as syringes, catheters, lancets, and test strips, 100% inspection is often impractical or impossible. Lot release therefore relies on Acceptance Sampling Plans.Acceptance sampling plans are fundamentally statistical hypothesis tests formulated as risk management tools. Under ISO 2859-1, the operating characteristics of an attribute sampling plan are defined by the Acceptance Quality Limit (AQL), which protects the producer from rejecting good lots. However, in regulatory validation, the focus shifts entirely to the consumer-risk end, characterised by the Limiting Quality (LQ) in ISO 2859-2 or the Rejectable Quality Limit (RQL). The RQL represents the defective rate that conventionally must be rejected with high probability (typically 95%).One of the most misunderstood frameworks is the Zero Acceptance Number plan (c=0), developed by Squeglia as an alternative to the MIL-STD-105 / ANSI/ASQ Z1.4 family, matched at the Limiting Quality point. In a c=0 plan, if a single defective unit is found in a sample of size n , the entire lot is rejected. The statistical derivation is elegant and inherently consumer-protective. Under a binomial model, the probability of accepting a lot with a true defective rate of p_0 is:P(\text{Accept}) = (1 - p_0)^nIf the objective is to be P confident that the lot has a defective rate no higher than p_0 , the required sample size is derived algebraically from the binomial Cumulative Distribution Function (CDF):n = \text{trunc}\left[ \frac{\ln(1 - P)}{\ln(1 - p_0)} \right] + 1For example, to achieve 95% confidence (P = 0.95) that the true defective rate is less than 5% ( p_0 = 0.05 ), the required sample size is n = 59 . This is not an arbitrary number; it is the algebraic consequence of the binomial probability model under a zero-acceptance criterion. This sample size also corresponds to the exact 95% upper confidence bound when zero failures are observed in 59 units (4.951%). When a Notified Body reviewer questions the sample size for biocompatibility testing or bench validation, the defence must be presented in these exact terms.For continuous data, Variables Sampling Plans based on Cpk (Process Capability Index) are frequently employed. Pardo (2023) highlights that the sampling distribution of \hat{Cpk} is related to the non-central t-distribution. However, a critical statistical blind spot occurs when engineers fail to translate the consumer-protective c=0 logic into the continuous Cpk framework.The hypothesis test for validation must be structured to protect the patient, treating the target capability as the RQL—the absolute minimum acceptable limit:H_0: Cpk \leq K_0 \quad \text{vs.} \quad H_1: Cpk > K_0To achieve 95% confidence that the true population Cpk exceeds K_0 , the critical value c for the sample Cpk must be determined such that if the true process is sitting exactly at the target capability ( Cpk_{\text{true}} = K_0 ), the probability of passing the test is only \alpha (e.g., 5%):\Pr\left[\hat{Cpk} \geq c \mid Cpk_{\text{true}} = K_0, n\right] \approx \alphaIf a device has a true population Cpk of 1.0 (corresponding to approximately 99.73% of units within specification limits, assuming a centred process), and the sample size is n = 36 , applying this consumer-protective framework means the critical value for the sample Cpk must be set at approximately 1.27. Under this stringent criterion, if the true Cpk is exactly 1.0, the probability of the sample passing the test is only 5%. This correctly places the statistical protection on the regulator/consumer, ensuring a marginally capable process will not easily pass.This counter-intuitive result is a direct consequence of sampling variability and the non-central t-distribution. It is one of the most common sources of validation failure in the medical device industry. Engineers often naively set the pass/fail criterion at Cpk = 1.0. If the criterion is set at 1.0, the probability of passing, even with a process sitting exactly at the target capability, is only approximately 52.7%. This provides essentially no regulatory assurance that the true Cpk is actually 1.0.Conversely, statistical textbooks sometimes demonstrate the maths using a producer-protective framework (setting the criterion at 0.8175 so a process at 1.0 passes 95% of the time). Applying this to regulatory validation is a fatal flaw: it allows a process with a true Cpk as low as 0.63 to pass 5% of the time. The naive criterion of 1.0 sits between the producer-protective (0.8175) and consumer-protective (1.27) critical values, providing no statistical assurance to either party (52.7% pass rate).Statistical Blind Spot 3: Equivalence Testing in the Absence of Codified MarginsStandard biostatistics training covers bioequivalence (BE) testing for generic drugs. The Two One-Sided Tests (TOST) procedure (Schuirmann, 1987) is used to demonstrate that a generic drug’s pharmacokinetic parameters (AUC, Cmax) are within 80–125% of the reference listed drug. The equivalence limits (0.80 and 1.25) are codified in regulation.In medical devices, equivalence is mandated by MDR Article 61 for clinical evaluation, but there are no codified statistical limits. The manufacturer must demonstrate “substantial equivalence” to a predicate device: a concept that requires both clinical and statistical defence.The fundamental statistical error in device equivalence testing is the use of standard superiority testing to infer equivalence. A conventional two-sample t-test with hypotheses:H_0: \mu_1 = \mu_2 \quad \text{vs.} \quad H_1: \mu_1 \neq \mu_2Failing to reject H_0 does not prove the devices are equivalent. It simply indicates that the sample size was insufficient to detect a difference. This is the “absence of evidence is not evidence of absence” fallacy, and it is a frequent cause of CER rejection by Notified Bodies.The correct formulation is the TOST framework:H_0: |\mu_1 - \mu_2| > \Delta \quad \text{vs.} \quad H_1: |\mu_1 - \mu_2| \leq \DeltaHere, \Delta is the equivalence margin: a pre-specified, clinically and functionally justified threshold beyond which a difference would be considered meaningful. The TOST procedure conducts two one-sided tests at level \alpha (not \alpha/2 ):Test 1: H_{0a}: \mu_1 - \mu_2 \leq -\Delta vs. H_{1a}: \mu_1 - \mu_2 > -\DeltaTest 2: H_{0b}: \mu_1 - \mu_2 \geq +\Delta vs. H_{1b}: \mu_1 - \mu_2 < +\DeltaBoth null hypotheses must be rejected to declare equivalence. The critical values use the 100(1-\alpha) th percentile of the t-distribution (not 100(1-\alpha/2) ), which is the key to the TOST procedure’s validity.The margin \Delta is not a statistical decision, it is an engineering and clinical decision. It may be based on the smallest detectable change of a measurement instrument, a percentage of the SOTA mean, or a clinically meaningful threshold. However, once \Delta is set, the sample size calculation must use the non-central t-distribution, accounting for the fact that the power of an equivalence test is maximised when the true difference is zero and decreases as the true difference approaches \Delta .For binary outcomes (e.g., pass/fail rates, sensitivity, specificity), the sampling distributions of differences in proportions and odds ratios are difficult to approximate reliably in small samples. In these cases, bootstrap resampling methods are required to compute percentile-method confidence intervals for the difference in proportions or the odds ratio, and to verify that these intervals fall entirely within the pre-specified equivalence bounds. Pardo (2023) provides detailed R implementations for bootstrap TOST confidence intervals, noting the lack of a simple closed-form sampling distribution as the primary rationale for this approach.Bridging Reliability Engineering and Survival AnalysisOne of the most profound areas of divergence between pharmaceutical and medical device biostatistics is the treatment of time-to-event data.In pharma, Kaplan-Meier curves and Cox regression are used to model patient survival. In medtech, while these methods are certainly used for patient survival endpoints (e.g., time to Major Adverse Cardiovascular Events for a stent), the same statistical foundations are uniquely applied to device survival. Frequently this occurs under accelerated conditions where the physics of failure must be incorporated into the statistical model.Pardo (2023) maps the first-order chemical kinetics model, which governs phenomena such as polymer degradation, battery depletion, reagent instability, and drug-elution profiles, to the exponential reliability function:R(t) = \Pr{T \geq t} = e^{-\lambda t}Medical devices rarely exhibit a constant failure rate. The “bath-tub” hazard curve is the standard model for hardware reliability. This is due to high early failure rates (due to manufacturing defects), a low constant failure rate during useful life, and an increasing failure rate during wear-out. The cumulative hazard function H(t) = -\ln(R(t)) and the empirical reliability function \hat{R}(t_k) = (n-k)/n provide the foundation for non-parametric reliability estimation.To predict a 2-year shelf life for an IVD reagent or a 10-year lifetime for an implantable, accelerated life testing (ALT) is mandatory. The Arrhenius reaction rate law relates the failure rate at an elevated temperature to the failure rate at nominal conditions:\lambda_P = A \exp\left(\frac{-B}{P}\right)Where P is temperature in Kelvin, B is the normalised energy of activation, and A is a material-specific constant. By testing devices at multiple elevated temperatures, the acceleration factor k can be estimated, and the failure rate at nominal storage conditions can be extrapolated.The statistical challenge is threefold:Censoring: ALT data are typically Type I censored (testing stops at a fixed time T_max). The Kaplan-Meier estimator or Maximum Likelihood Estimation (MLE) with right-censoring must be used to handle the fact that not all units will have failed by the end of the test.Model validity: The Arrhenius model assumes that the failure mechanism at elevated temperature is identical to the failure mechanism at nominal temperature. This assumption must be tested – not assumed. Diagnostic checks on the residuals of the accelerated failure time model, and examination of the failure mode of each censored unit, are essential.Design integration: Pardo’s approach of fitting polynomial approximations to G(t) = −ln F(t) (the negative log of the failure-time distribution) at each combination of design factors, and then using least squares to relate the polynomial coefficients to those factors, allows reliability optimisation without assuming a specific parametric failure distribution. This is particularly powerful for implantable devices where the failure mode may be a complex function of material, geometry, and biological environment.The Statistician as a Regulatory StrategistThe role of the biostatistician in the medical device industry extends far beyond the execution of statistical tests. The value lies in acting as a translator and strategist across the entire product lifecycle:Design Control (ISO 13485 7.3): During product design, advocacy for Design of Experiments (DOE) – specifically fractional factorial or Central Composite Designs (CCDs) – enables efficient optimisation of device parameters. The alternative, “one-factor-at-a-time” experimentation, is both statistically inefficient and incapable of detecting interaction effects between design factors.Risk Management (ISO 14971:2019): Risk files frequently rely on arbitrary 1–5 scales for occurrence and severity in FMEA. These can be replaced with quantified probabilities derived from historical complaint data, tolerance intervals, and Bayesian updating. This transforms the risk file from a qualitative exercise into a defensible statistical document.Clinical Evaluation and Investigation (MDR Annex XIV & XV): The clinical evaluation plan (Annex XIV) and clinical investigation plan (Annex XV) must include a statistically justified methodology for literature appraisal, SOTA benchmarking, and equivalence demonstration. The statistical section of the Clinical Evaluation Report (CER) is one of the most heavily scrutinised elements of a technical documentation file.Post-Market Surveillance (MDR Article 83): PMS systems require statistical control charts (CUSUM, EWMA) to detect drift in real-world data. Periodic Safety Update Reports (PSURs) must contain proactive, statistically significant signal detection – not passive complaint listing. The use of autoregressive integrated moving average (ARIMA) models and Markov chains for state transition modelling in post-market data is an emerging area of regulatory expectation.The transition from academic biostatistics to medical device regulatory strategy requires a fundamental re-conceptualisation of when, why, and how statistical methods are applied. The methods themselves – tolerance intervals, acceptance sampling, TOST, accelerated failure time models, mixed-effects VCA, bootstrap resampling – are not new. Their application within the MDR/IVDR/UKCA regulatory framework, across a hybrid ecosystem of engineering, analytical, clinical, and real-world data, is what distinguishes a medtech biostatistician from a clinical trials statistician.Sources: GOV.UK – Regulating medical devices in the UK, MHRA consultation on indefinite CE recognition (Feb 2026),EC proposal to simplify MDR/IVDR (Dec 2025), IVDR transition periods 2026, ISO/CD 15197 (revision in draft).Pardo, S. A. (2023). Statistical Methods and Analyses for Medical Devices. Springer.European Parliament and Council. (2017). Regulation (EU) 2017/745 on medical devices (Medical Device Regulation – MDR). Official Journal of the European Union.European Parliament and Council. (2017). Regulation (EU) 2017/746 on in vitro diagnostic medical devices (IVDR). Official Journal of the European Union.European Commission. (2016). MEDDEV 2.7/1 revision 4: Guidelines on medical devices clinical evaluation.Medical Device Coordination Group (MDCG). (2020). MDCG 2020-5: Clinical Evaluation – Equivalence.Medical Device Coordination Group (MDCG). (2020). MDCG 2020-6: Clinical Evidence Needed for Medical Devices.Medical Device Coordination Group (MDCG). (2020). MDCG 2020-13: Clinical Evaluation Assessment Report Template.International Organization for Standardization. (2016). ISO 13485:2016 – Medical devices – Quality management systems.International Organization for Standardization. (2019). ISO 14971:2019 – Medical devices – Application of risk management to medical devices.International Electrotechnical Commission. (2015). IEC 62366-1:2015 – Medical devices – Application of usability engineering to medical devices.International Organization for Standardization. (2014). ISO 16269-6:2014 – Statistical interpretation of data – Part 6: Determination of statistical tolerance intervals.

IntroductionUnder the EU Medical Device Regulation (MDR) and the In Vitro Diagnostic Regulation (IVDR), the scrutiny applied to clinical evidence has fundamentally shifted. Notified Bodies and the UK MHRA are no longer simply checking for the existence of clinical data; they are interrogating the methodological soundness of how that data is generated, analysed, and interpreted.At the centre of this regulatory attention is the Statistical Analysis Plan (SAP). The SAP is a binding, operational execution manual that translates clinical hypotheses into evidence generating methodology. Rather than a static theoretical blueprint, it is an active document that forms the methodological foundation of the Clinical Evaluation Report (CER) or Performance Evaluation Report (PER). For study Statisticians and Statistical Programmers, the SAP has a critical operational function: It serves as a formal workflow document that standardises how the statistician and programming team process study data from database lock to final reporting. It defines the sequential steps for data handling, outlines the precise logic for analysis, and sets the technical specifications for producing final tables and listings. By establishing this structured workflow before unblinding, it ensures the entire evaluation process is objective, consistent, repeatable and fully auditable.What is needed to draft a SAP?A robust SAP cannot be drafted in a vacuum. A statistician requires a comprehensive stack of finalised, or near-finalised, source documents. Without them, the SAP will inevitably contain contradictions, miss critical derivations, or misdefine analysis cohorts.It is equally important to understand the direction of the clinical workflow. Certain documents are prerequisites that feed into the SAP, while other critical operational documents are downstream outputs driven by the SAP.Before SAP drafting begins, the author must integrate the following inputs:1. The final study protocol (QC’d by the clinical team, minus the statistical sections)The clinical sections of the study protocol are non-negotiable prerequisites. The statistician requires the finalised clinical objectives, the exact definitions of the endpoints, and the inclusion and exclusion criteria. The clinical team must QC and lock these clinical elements before the SAP can be drafted. The statistical sections within the study protocol itself are often high-level and are frequently refined concurrently with, or driven directly by, the granular specifications of the SAP. Attempting to write a SAP while the clinical narrative of the study protocol is still shifting guarantees endless rewrites.2. The annotated (electronic) Case Report Form (eCRF) and completion guidelinesA blank Case Report Form marked up with the exact database variable names, question text, and valid data ranges for every data point collected at the clinical sites.The eCRF is the absolute blueprint of how data is captured. A statistician cannot define derivations without knowing the exact structure of the incoming data. If a SAP specifies an analysis of “change from baseline in vessel diameter,” but the eCRF only captures categorical data, the SAP is fundamentally flawed. The eCRF grounds the statistical methodology in the reality of data collection and enables the code for analysis and TLFs to be developed.3. The official Data Management Plan (DMP)The DMP outlines how data will be cleaned, coded, and handled. The statistician needs to understand the rules for data cleaning, as this impacts how missing, partial, or implausible data will be treated in the analysis. It also defines important derivations, such as how adverse events are coded using standard dictionaries like MedDRA. 4. Instructions for use (IFU) and reference standard algorithmFor medical devices, the IFU defines the intended use population, which dictates the primary analysis cohort. For IVDs under IVDR, the algorithm defining how the “truth” is adjudicated is critical. The statistical analysis of sensitivity and specificity is entirely dependent on the integrity of this algorithm.5. Data standardisation frameworks (if applicable) If the study is utilising CDISC standards (such as SDTM and ADaM), which many global MedTech companies adopt for data harmonisation or to support concurrent FDA submissions, the statistician requires these implementation guides as inputs. This ensures the SAP derivations map correctly to standard regulatory data structures. If CDISC is not being used, the SAP cannot simply ignore dataset architecture. In the absence of external standards, the SAP must explicitly define the structure of the analysis datasets, the naming conventions for variables, and the metadata. Whether using CDISC or a proprietary structure, the SAP must provide enough detail to guarantee that the resulting datasets are reproducible and auditable by a Notified Body.Conversely, certain critical operational documents cannot be finalised until the SAP is designed, as they are direct results of the statistical methodology:Mock tables, listings, and figures (TLF) shells: The SAP dictates the exact analysis populations, derivations, and statistical methods required. The mock TLF shells, which define the precise layout and data content of the final outputs, must be built directly from these SAP specifications. Attempting to finalise mock shells before the SAP is locked is a fundamental workflow error.IRT specifications and randomisation schedule: While the study protocol dictates the high-level design, the exact stratification factors, allocation ratios, and randomisation methodologies are operationally finalised in the SAP. The interactive response technology system (IRT) is subsequently built to these SAP specifications, and the randomisation schedule is generated as a direct result.Statistical sections of the Clinical Study Protocol: These sections will be derived from and consist of a highly abridged version of the advanced SAP draft based on the final pre- stats version of the protocol. The “locking” of the SAP itself often occurs after this, coinciding with database lock.Designing the document for the persons doing the workThe primary functions of a SAP are to ensure scientific integrity, prevent data-driven inference, and guarantee reproducibility. However, from the perspective of the statistician and programmer executing the analysis, the document must also serve a highly practical function: it must facilitate an efficient, error-minimising workflow.In theoretical document design, there is a general rule to avoid redundancy – the “Don’t Repeat Yourself” principle. The idea is that a global definition should be stated once, and subsequent sections should simply cross-reference it. When applied to a complex, multi-page SAP, this creates a significant operational problem.If a statistical programmer is writing code for a specific analysis, but the SAP forces them to hold a complex rule in their head, scroll up forty pages to check the definition of the target population, scroll to another section to check the covariates, and scroll back to find their place, the document has failed them. This constant context-switching causes cognitive fatigue and is a primary driver of errors.The alternative is to use local context. Rather than relying on distant cross-references, a well-designed SAP makes each analysis section self-contained.Instead of writing: “Analysis of Primary Safety will be performed as specified in Section 4.2 using the population defined in Section 3.1,” the document explicitly restates the applicable rules within the local section:“Analysis of Primary Safety: The Safety Analysis Set will be used, defined as all subjects who underwent the index procedure with the investigational device. The primary composite safety endpoint at 30 days will be analysed using the Kaplan‑Meier method; the survival proportion at 30 days and its 95% confidence interval will be estimated using the log‑log transformation to ensure the interval remains within [0, 1]. Subjects lost to follow‑up prior to day 30 will be censored at their last known alive date.”By intentionally restating the applicable rules within the context of each specific analysis, the document offloads the burden from the programmer’s working memory. The theoretical risk of having to update text in multiple places if a global definition changes is minor compared to the very real risk of errors caused by cognitive overload. This self-contained structure is not solely for the benefit of writing code. When the statistician is conducting the analysis, interpreting the outputs, and drafting the clinical or performance evaluation report, they need to instantly recall the exact methodological rules applied to that specific endpoint – such as how missing data were handled or which covariates were included. A document that forces the author to hunt through distant sections to reconstruct the analytical context disrupts the flow of interpretation just as severely as it disrupts coding. A SAP should be optimised for the humans executing the work and writing the narrative, not for the theoretical elegance of the document architect. What appears like a repetative or overly reduncant document is oftentimes set-up that way to acheve workflow efficiency.The necessity of senior statistical reviewBecause the SAP serves as the execution manual, its quality control (QC) process cannot be treated as a simple editorial review. A junior statistician may draft a grammatically flawless SAP, but a senior statistician is required to perform methodological QC.This is particularly important in the EU/UK MedTech space. Sample sizes are often smaller than in pharmaceutical trials, and data structures are inherently more complex, such as clustered data from multiple lesions per subject, or repeated measurements over time from a single sensor. A senior statistician has the experiential foresight to identify methodological traps before any code is written.When reviewing a device or diagnostic SAP, a senior statistician will review the document as a whole in addition to probing several specific areas (not an exhaustive list):Model feasibility: Will a proposed model converge if the device trial has a small sample size with an average cluster size of only 1.2 lesions per subject?Diagnostic accuracy: For IVDs under IVDR, does the proposed analysis for sensitivity and specificity adequately account for subjects with indeterminate or missing reference standard results? Are the confidence intervals for the Receiver Operating Characteristic (ROC) curve calculated using the appropriate methodology (e.g., DeLong, Obuchowski) for the study design?Handling of missing data: Beyond the high-level estimand framework, are the primary and sensitivity analyses robust against the expected missing data patterns? Does the plan appropriately differentiate between Missing at Random (MAR) and Missing Not at Random (MNAR) assumptions, and are the sensitivity models (e.g., multiple imputation, tipping point analysis) specified correctly?Bayesian and adaptive methodologies: If the device trial leverages historical data or adaptive designs, is the selection of prior distributions statistically defensible and clearly pre-specified? Are the adaptive decision boundaries calculated in a way that controls the Type I error?Interim analysis integrity: If planned interim looks are specified, are the stopping boundaries and alpha-spending functions correctly defined to control the overall Type I error rate without introducing operational bias?Regulatory alignment: Does the handling of intercurrent events, such as device malfunctions or procedure deviations, align with the estimand framework outlined in ICH E9(R1)?Notified Body scrutiny: Will the proposed multiplicity strategy withstand the statistical review of a Notified Body operating under MDR Annex XIV?Analysis set definitions: Are the precise rules for allocating subjects to the Intention-to-Treat, Per-Protocol, and Safety sets unambiguously defined? For instance, is it explicitly clear how to handle a subject who was randomised but underwent a different procedure, or a subject with no post-baseline data?Visit windowing and derivations: Are the derivation rules for calculating baseline, endpoints, and change-from-baseline explicitly clear? Do the defined visit windows (e.g., Day 30 ± 7 days) align logically with the data collection schedule, and are the rules for selecting data points within overlapping windows (e.g., “closest to the target day,” “worst case”) specified?Output specifications and denominators: Do the mock table shells perfectly align with the SAP text? Are the exact rules for calculating percentages and N-counts explicitly defined (e.g., “N = number of subjects in the Safety Set; n = number of subjects with the event”) to prevent programming ambiguity?Safety event counting rules: How are recurrent adverse events handled: by subject or by event? How are MedDRA coded terms grouped (e.g., System Organ Class vs. Preferred Term) for presentation, and how are pre-existing conditions differentiated from new-onset events?Continuous vs. categorical data: If a continuous variable is being dichotomised (e.g., “success” if vessel diameter > 2.0 mm), is the threshold clearly defined, and is the handling of missing or indeterminate continuous values specified before the categorical transformation is applied?Discovering a statistical flaw during the drafting of a clinical evaluation report is highly disruptive, often requiring a complete re-analysis of the dataset, delayed regulatory submissions, and significant financial cost. Senior methodological review acts as the necessary safeguard to prevent these flaws from entering the execution phase.The boundaries of non-statistical reviewTo streamline operations, clinical sponsors occasionally route the SAP for primary QC to a clinical project manager, medical monitor, or clinical affairs lead. This cross-functional review is highly valuable. Non-statisticians play a crucial role in verifying clinical alignment, ensuring that endpoint definitions perfectly match the study protocol and that the clinical narrative is accurately reflected. However, the boundaries of methodological review must be clearly delineated to protect the integrity of the document.When a clinical reviewer evaluates the mathematical parameters of a SAP, there is a risk of a “false pass.” A complex sentence specifying a mixed-effects model for repeated measures may read grammatically well and appear clinically reasonable. However, without deep statistical specialisation, it is difficult to assess whether that model is over-parameterised for the expected data structure and will ultimately fail to converge.The deliberate repetition of rules throughout sections- designed to prevent context-switching during programming – may appear to a non-statistician as an editorial oversight. This can lead to reviewers wasting valuable time commenting on and attempting to “correct” this structural redundancy, pulling focus from the clinical alignment they are actually there to verify.This crossover between clinical and statistical expertise can inadvertently trigger the Dunning-Kruger effect. When operating outside our core domain, human nature often prevents us from seeing the methodological complexities we do not understand. Consequently, well-intentioned clinical reviewers might suggest methodological changes that are clinically logical but technically or mathematically unsound.This is not a reflection of clinical capability, but a natural consequence of crossing disciplinary boundaries.When a “false pass” or a methodologically compromised plan proceeds to the programming phase, the code will often execute without generating a technical error. The true detriment emerges later: the analysis will reliably produce statistically invalid, biased, or uninterpretable outputs that look like genuine results. The rework required at this late stage- revisiting the SAP, reprogramming the datasets, and rewriting the clinical evaluation report- is exponentially more expensive and time-consuming than catching the methodological error during the review phase.Evaluating AI in statistical QCAs artificial intelligence (AI) and large language models become integrated into clinical operations, there is a temptation to use AI for SAP QC. While AI can be helpful, it must be approached with caution. AI functions as a syntactic engine, not a mathematical authority. It can perform administrative QC, but it is incapable of nuanced methodological QC. In certain ways the “intelligence” of artificial intelligence is a misnomer or at least greatly overestimated.On the administrative side, AI is highly valuable. It can cross-reference the SAP against the study protocol to flag missing endpoints, detect internal semantic contradictions, identify missing standard sections, and standardise terminology. For a first-pass cleanup, AI is a useful tool.However, AI often fails when asked to evaluate methodology. It cannot assess whether a proposed statistical model will converge given the expected sample size and data sparsity of a specific clinical trial. It reads the text and recognises a valid statistical method, but it cannot project the mathematical reality of the data. AI also may lack access to the nuanced, unpublished precedents of EU Notified Bodies or the MHRA. It provides textbook statistical answers, which are frequently misaligned with the specific regulatory strategy required for a successful MDR or IVDR submission. When presented with complex adaptive designs, AI is prone to confidently “correcting” the text based on standard principles, thereby corrupting the methodology.There is also the matter of cost. With clinical trial data being highly sensitive, LLM use via standard consumer APIs is not appropriate. The infrastructure costs of a locally hosted (or AWS cloud hosted) LLM – needed to ensure the protection of intellectual and commercial property – can often outpace the hourly rate of an experienced statistician. This is because human oversight is required to prompt the QC process, QC the QC and iterate when things inevitably fall short due to inherent limitations. Anecdotally, attempts to incorporate AI have in many cases lead to a task that is completed in more time, not less.SummaryIn the highly regulated EU and UK medical device landscape, the SAP is a critical component of clinical evidence. It is an operational manual that must balance regulatory rigour with practical efficiency. A well-constructed SAP protects the statistician’s working memory through local context, supporting both the coding and the clinical writing phases. It relies on the methodological foresight of a senior statistician to navigate the mathematical complexities of device and diagnostic data. It recognises the boundaries of non-statistical review and the limitations of artificial intelligence. Finally, it is built upon a solid, integrated foundation of prerequisite clinical documentation, while driving the operational outputs that follow. Treating the SAP with the operational and strategic respect it deserves is essential for a smooth regulatory submission and successful market access.References:ICH E9(R1): ICH. (2019). ICH E9(R1): Addendum on estimands and sensitivity analysis in clinical trials to the guideline on statistical principles for clinical trials. European Medicines Agency.ICH E6(R3): ICH. (2025). ICH E6(R3): Good Clinical Practice. Adopted 6 January 2025.MDR 2017/745: European Parliament and Council. (2017). Regulation (EU) 2017/745 on medical devices (MDR). Official Journal of the European Union.IVDR 2017/746: European Parliament and Council. (2017). Regulation (EU) 2017/746 on in vitro diagnostic medical devices (IVDR). Official Journal of the European Union.FDA Guidance on Adaptive Designs: FDA. (2016). Adaptive Designs for Medical Device Clinical Studies: Guidance for Industry and FDA Staff. FDA.CDISC SDTM/ADaM: CDISC. (2021). Study Data Tabulation Model (SDTM) Implementation Guide. CDISC. / ADaM Implementation Guide. CDISC.MedDRA: International Council for Harmonisation. (2023). MedDRA® terminology. ICH.

Medical device clinical investigations, as formally regulated under FDA Investigational Device Exemption (IDE) rules or EU MDR/ISO 14155 guidelines, increasingly incorporate adaptive methodologies to improve efficiency and patient outcomes. Two primary approaches have emerged: Multi-Arm Multi-Stage (MAMS) designs, which evaluate arms at predefined milestones to drop inferior treatments or stop early for efficacy, typically maintaining a fixed allocation ratio among remaining arms; and Response-Adaptive Randomisation (RAR), which continuously shifts the randomisation probabilities of future patients toward better-performing treatments throughout the investigation.The choice between these methods depends on investigation characteristics including endpoint timing, data infrastructure, regulatory requirements, and scientific objectives. This guide examines the mathematical foundations and operational considerations for each approach to help biostatisticians and sponsors select the most appropriate method.Fundamental Differences Between MAMS and RARThe core distinction lies in the mechanism of adaptation. MAMS designs adapt by eliminating underperforming arms or stopping the investigation entirely at discrete interim analyses. Between these analyses, randomisation remains fixed (e.g., 1:1 among active arms).Conversely, RAR designs adapt the allocation ratio itself. Rather than dropping arms, RAR continuously updates the probability of assignment, skewing enrolment toward the treatment arms demonstrating the best outcomes. This fundamental difference has profound implications for statistical methodology, operational complexity, and regulatory considerations.FeatureMAMSRARAdaptation mechanismDrops arms at discrete interim analysesContinuously shifts allocation ratiosRandomisation ratioFixed between interim analysesContinuously updatedEndpoint timingTolerant of delayed endpointsRequires rapid endpoint assessmentType I error control\alpha-spending plus multi-arm adjustmentUsually simulation-based calibrationData infrastructurePeriodic clean data cuts sufficientReal-time data systems requiredTemporal drift vulnerabilityModerateHighPower at fixed sample sizeStable across scenarios, but still requires simulation to characteriseVariable; may gain or lose depending on target allocationEstimation after adaptationBiased; established correction methodsBiased; correction methods less standardisedRegulatory familiarityHigh (group sequential framework)Moderate, with specific reservationsMulti-Arm Multi-Stage (MAMS): Mathematical Framework and ImplementationMAMS designs adapt through formal interim analyses. The mathematics centres on group sequential testing, where investigators evaluate treatment effects and apply pre-specified stopping rules for efficacy or futility.Let H_{0,i} denote the null hypothesis for treatment i against control. This hypothesis is tested at each of the interim analysis stages k = 1, 2, \ldots, K using test statistics Z_{i,k}, which are compared against predefined boundaries:If Z_{i,k} \geq U_k, stop for efficacy and reject H_{0,i}If Z_{i,k} \leq L_k, drop treatment i for futilityIf L_k < Z_{i,k} < U_k, continue enrolling 1:1 in remaining armsThe Lan-DeMets approach allocates Type I error across K analyses using \alpha(t_k) = f(t_k), where t_k = I_k / I_{\max} is the information fraction and f(t) is the spending function. For the O’Brien-Fleming boundary:f(t) = 2 - 2\Phi!\left(\frac{z_{1-\alpha/2}}{\sqrt{t}}\right)This function applies stringent boundaries early in the investigation when data are sparse, reaching \alpha at t = 1. Note that f(t) returns the cumulative error spent by information fraction t; the increment available at stage k is f(t_k) - f(t_{k-1}).Multiple comparisons present a dual challenge. Lan-DeMets \alpha-spending controls the Type I error rate across repeated interim looks, but a separate adjustment is needed for the multiplicity of K experimental arms. Because the test statistics are correlated through the shared control arm, a Bonferroni correction is unnecessarily conservative. The standard approach for MAMS designs is the generalised Dunnett test (Magirr, Jaki & Whitehead, 2012), which accounts for both the number of arms and the number of stages. It is worth noting that whether family-wise error rate control is even required in multi-arm investigations remains genuinely contested; regulators have varying expectations depending on the confirmatory or exploratory nature of the investigation.Worked Example: Cardiac Stent InvestigationConsider a cardiac stent investigation testing three new drug-eluting stents (A, B, C) against standard care (D). After enrolling 200 patients (1:1:1:1 fixed allocation), the first interim analysis is conducted. Stent A shows a 15% reduction in target vessel revascularisation, while Stents B and C perform similarly to the control.Using an O’Brien-Fleming spending function with a Dunnett-based multi-arm adjustment, each stent is compared against standard care against an efficacy boundary U_1 and a futility boundary L_1. These are derived at the design stage from the number of arms, the number of planned analyses, and the information fraction at each.The interim decision applies a separate rule to each arm, so more than one of the following can occur at the same analysis:If Stent A crosses U_1, the investigation stops early for overwhelming efficacy.If Stents B and C fall below L_1, they are dropped, and the investigation continues with fixed 1:1 allocation between Stent A and Standard Care.If neither boundary is crossed, the investigation continues with 1:1 allocation among all remaining arms.Note that with a 12-month target vessel revascularisation endpoint, the number of patients with mature outcomes at the point 200 are enrolled may be considerably smaller than 200. The information fraction, not the enrolment fraction, drives the boundary calculation.This approach requires sophisticated upfront planning. Sponsors must specify exactly when interim analyses occur, the exact boundaries for dropping arms, and how Type I error is controlled. Regulatory agencies appreciate this level of pre-specification because it ensures investigation integrity and prevents ad-hoc decision-making.Response-Adaptive Randomisation (RAR): Bayesian and Frequentist ApproachesResponse-adaptive randomisation operates at a much more granular level, updating beliefs about treatment effectiveness as patient outcomes accrue. Rather than dropping arms, RAR shifts allocation probabilities. The mathematical foundation typically involves Bayesian updating, where the investigation maintains probability distributions representing current beliefs about each treatment’s efficacy.Thompson sampling maintains posterior distributions for each treatment’s efficacy parameter. For binary outcomes, if treatment i has observed s_i successes in n_i trials, the posterior under a \mathrm{Beta}(\alpha, \beta) prior becomes:\theta_i \mid \text{data} \sim \mathrm{Beta}!\left(\alpha + s_i,; \beta + n_i - s_i\right)At each allocation, a value \tilde{\theta}_i is sampled from each posterior, and the next patient is assigned to the treatment with the highest sampled value. The allocation probability for treatment i is therefore:\pi_i = P!\left(\theta_i = \max_j \theta_j ;\middle|; \text{data}\right)This is the posterior probability that treatment i is the best among the available options, and it holds exactly by construction rather than as a limiting result.In practice, raw Thompson sampling is rarely used unmodified in confirmatory settings. Tempering the allocation probabilities, raising them to a power c \in (0,1) and renormalising, moderates the skew and improves power. Protecting the control allocation at a fixed proportion is also common, and is often what makes RAR viable in a multi-arm investigation at all.Allocation probabilities can alternatively be driven by posterior means. Because posterior means do not sum to one across arms, they must be normalised:\mu_i = \frac{\alpha_i + s_i}{\alpha_i + \beta_i + n_i}, \qquad \pi_i = \frac{\mu_i}{\sum_{j=1}^{K} \mu_j}A frequentist alternative is the Randomised Play-the-Winner (RPW) rule of Wei and Durham (1978), which uses an urn model rather than a direct probability calculation. The urn begins with u balls of each treatment type. A ball is drawn and the corresponding treatment assigned; a success on treatment A adds w balls of type A, while a failure on treatment A adds w balls of type B. The per-patient assignment probability is governed by the current urn composition, which converges to a limit determined by the failure rates q_i = 1 - p_i:\lim_{n \to \infty} \frac{n_A}{n} = \frac{q_B}{q_A + q_B}This allocates more patients to the treatment with the lower failure rate. For example, if the true failure rates are q_A = 0.2 and q_B = 0.4, the limiting allocation to treatment A is 0.4 / 0.6 \approx 67%. This differs from the simpler success-proportion heuristic p_A / (p_A + p_B) \approx 57%, which is sometimes described as RPW but is a different rule with different asymptotic behaviour.Additional RAR rules include Covariate-Adjusted Response-Adaptive (CARA) randomisation, which models success probability as:\operatorname{logit}!\big(P(Y = 1 \mid X, Z)\big) = X^{\mathsf{T}}\beta + Z^{\mathsf{T}}\gamma + (X \otimes Z)^{\mathsf{T}}\deltawhere X are baseline covariates and Z indicates treatment assignment. The interaction term \delta is what makes this a CARA model rather than a covariate-adjusted analysis: without it the treatment effect is common to all patients and there is nothing for allocation to condition on.For continuous outcomes following N(\mu_i, \sigma^2), assuming known variance and a flat prior, Thompson sampling updates:\mu_i \mid \text{data} \sim N!\left(\hat{\mu}_i,; \frac{\sigma^2}{n_i}\right)where \hat{\mu}_i is the sample mean for treatment i. If \sigma^2 is unknown, the marginal posterior follows a t-distribution and a prior on \sigma^2 must be specified; the normal form above is then an approximation that is poor at small n_i.Worked Example: AI Diagnostic System InvestigationConsider a clinical investigation testing three approaches for detecting diabetic retinopathy: AI-only, traditional ophthalmologist review, and combined AI plus ophthalmologist verification, with concordance against an adjudicated reference standard as the endpoint.Starting with uninformative priors \mathrm{Beta}(1,1) for each approach, after 50 patients per arm (150 in total) the data shows: AI-only correctly diagnosed 42 cases with 8 errors; traditional review got 38 right with 12 wrong; and the combined approach achieved 47 correct with only 3 errors. The Beta distributions become \mathrm{Beta}(43, 9), \mathrm{Beta}(39, 13), and \mathrm{Beta}(48, 4) respectively.For the next patient allocation, the system samples from each distribution many times and counts how often each approach produces the highest sample. The combined approach, with its dominant \mathrm{Beta}(48,4) distribution, wins approximately 94% of these samples, earning it an allocation probability of roughly 94% for the next patient under untempered Thompson sampling. No arm is dropped, but the AI-only and traditional review arms would receive very few future patients — which illustrates why tempering or control-arm protection is usually necessary if the investigation still needs to estimate the inferior arms with any precision.One design caveat deserves emphasis. Where the endpoint is purely diagnostic accuracy, all three strategies can in principle be applied to the same patient. A paired multi-reader multi-case design eliminates between-patient variability entirely and is substantially more efficient than randomising patients to reading strategies. Randomised allocation, and therefore RAR, is appropriate where the diagnostic strategy determines downstream clinical management and the endpoint is a patient outcome rather than a reading-level accuracy measure.The Delayed Response ProblemThis is where RAR is most fragile. Thompson sampling does not naturally handle delayed responses; the ability to update in real time depends on immediate endpoint assessment. When diagnostic results take weeks to confirm, beliefs cannot be immediately updated for patients enrolled yesterday, and the adaptation operates on a stale and progressively less representative subset of the data.Bai, Hu and Rosenberger (2002) addressed this theoretically, establishing the asymptotic properties of urn-based adaptive designs under delayed response. The broad finding is reassuring asymptotically: many adaptive designs retain their limiting allocation properties provided the degree of delay decays at a sufficient rate relative to sample size. This is cold comfort in a device investigation of a few hundred patients, where delay is a first-order practical problem rather than a vanishing higher-order term.The pragmatic implementation maintains separate pending pools for each treatment, applying the adaptation rule only to data from patients whose outcomes are confirmed available by time t - d. Note that this is a change of allocation rule, not merely a change of timing: allocation is now driven by normalised posterior means rather than by Thompson sampling. The posterior mean for each arm is calculated as:\mu_i(t) = \frac{\alpha_i + s_i(t-d)}{\alpha_i + \beta_i + n_i(t-d)}These posterior means are then normalised to form allocation probabilities:\pi_i(t) = \frac{\mu_i(t)}{\sum_{j=1}^{K} \mu_j(t)}where K is the number of active treatment arms, and s_i(t-d) and n_i(t-d) represent successes and total allocations from patients whose outcomes had matured by time t-d. More recent work extends this to group-sequential updating: Zhai, Li, Zhang and Hu (2024) propose RAR procedures that update on the responses of a group of patients, either as they become available or at fixed weekly or biweekly intervals, which maps far more naturally onto the operational reality of a device investigation than per-patient updating.Temporal Drift: A Critical RAR CautionA central concern with response-adaptive randomisation in the literature is temporal drift. Because allocation ratios change over calendar time, any drift in patient characteristics, clinical practice, or standard of care over the course of the investigation becomes confounded with treatment assignment. A treatment that happened to be favoured during a period of easier patients accumulates an advantage that randomisation would ordinarily have removed. The effect is not subtle and does not require bad faith; it follows mechanically from the design.This is compounded by a second problem. Early allocation decisions are made on the smallest and noisiest data, and their consequences persist. A run of good luck on one arm in the first twenty patients can skew allocation for the remainder of the investigation, and in the multi-arm case can starve the control arm precisely when it is most needed for comparison.Proschan and Evans (2020) caution against RAR in settings where temporal drift is plausible. In device investigations, where surgical techniques, imaging technologies, patient selection criteria, and the investigational device itself may all evolve during the trial, this risk is amplified.Mitigations exist and should be pre-specified: an initial burn-in period of equal randomisation before adaptation begins; protecting the control arm at a fixed allocation proportion; block-based rather than continuous updating; and adjusting for enrolment period in the final analysis. Saville and colleagues’ “Bayesian time machine” approach models temporal drift explicitly within the analysis rather than attempting to design around it.Power and Allocation VariabilityRAR is often presented as straightforwardly more efficient. The reality is more conditional, and the direction of the effect depends on what the allocation rule targets.In two-arm settings, response-adaptive randomisation frequently reduces power versus fixed 1:1 randomisation at a given total sample size. Rules targeting Neyman or optimal allocation can preserve or improve power, while rules targeting patient benefit — Thompson sampling among them — typically cost power, because power is maximised near balanced allocation and RAR deliberately moves away from it. There is a real trade-off between the number of patients who receive the better treatment during the investigation and the precision with which the investigation answers its question.Thompson sampling additionally has high variability in realised allocation. The expected allocation may favour the superior arm while individual realisations vary widely, and in a non-trivial fraction of investigations more patients end up on the inferior arm than the superior one. This variability is itself an operating characteristic that should be simulated and reported, not just the mean allocation.The multi-arm case is more favourable to RAR than the two-arm case. Wason and Trippa (2014) compared Bayesian adaptive randomisation directly against MAMS designs and found that RAR procedures can achieve higher power than MAMS when a single experimental treatment is genuinely effective — the most directly relevant published comparison for the choice this guide addresses.Device-Specific Statistical ConsiderationsMedical devices introduce mathematical challenges that pharmaceuticals rarely face.Learning CurvesLearning curves create a particularly complex problem because early poor outcomes might reflect operator inexperience rather than device inferiority. As described by Berry and colleagues (2010), this can be modelled explicitly. If \theta_{ij} represents the success probability for surgeon i on case j, a hierarchical model might use:\operatorname{logit}(\theta_{ij}) = \alpha_i + \beta_i \log(j) + \gamma Z_{ij} + \eta Z_{ij}\log(j)where \alpha_i captures surgeon i‘s baseline ability, \beta_i represents their learning rate, \gamma is the true device effect, Z_{ij} is an indicator of device assignment (0 for control, 1 for experimental), and \eta allows the experimental device to be learned at a different rate from the comparator. The logarithmic term captures the typical learning curve shape where improvement is rapid initially, then plateaus. The hierarchical structure is completed with prior distributions:\alpha_i \sim N(\mu_\alpha, \sigma^2_\alpha), \qquad \beta_i \sim N(\mu_\beta, \sigma^2_\beta)It is tempting to conclude that this framework makes RAR safe for devices by preventing promising devices from being penalised during skill acquisition. The opposite is closer to the truth.A learning curve is a temporal trend. It guarantees that early outcomes are systematically worse than later ones for reasons unrelated to the device. RAR adapts most strongly on exactly this early, operator-contaminated data, and its allocation decisions persist. Worse, the hierarchical model cannot rescue the situation in real time: reliable estimation of \beta_i requires sufficient cases per surgeon, and those cases do not exist at the point where RAR is making its most consequential allocation decisions. The model is a sound analysis strategy and a poor adaptation strategy.For device investigations with meaningful learning curves, this argues for MAMS, or for RAR with a substantial equal-randomisation burn-in extending past the steep portion of the curve. It also argues for pre-specifying a minimum case volume per operator before that operator’s data contributes to adaptation at all.Software Updates During InvestigationsSoftware updates present another mathematical puzzle. Instead of treating device versions as completely separate entities, hierarchical Bayesian methods model them as related. If version 1.0 has parameter vector \theta_1 and version 1.1 has \theta_2, the model can specify:\theta_2 \sim N(\theta_1 + \delta, \Sigma)where \delta represents expected improvement and \Sigma captures uncertainty about how modifications affect performance. Where \theta represents probabilities, this relationship should be specified on the logit scale rather than the natural scale. This approach borrows strength from pre-update data while learning about post-update performance.The statistical machinery, however, is downstream of a regulatory question that must be settled first: whether the updated device is still the device under investigation. A modification that changes intended performance characteristics may require a protocol amendment, a new risk assessment, or in some cases a separate investigation.Statistical Challenges and SolutionsData MaturityBoth MAMS and RAR amplify “garbage-in, garbage-out” problems because incorrect early decisions cascade through the entire investigation. The concern is more acute for RAR, which adapts continuously on immature data, than for MAMS, which typically adapts at discrete points against cleaner data cuts.For delayed outcomes, adaptation should be based only on mature data. The principled options are time-to-event modelling, multiple imputation, or restricting the adaptation rule to outcomes that have been confirmed available by the time of the next allocation decision. Ad-hoc down-weighting of immature observations should be avoided: for a binary endpoint, an observation that has not yet reached its follow-up window is censored rather than a partially-informative zero, and weighting schemes applied to the numerator alone attenuate confirmed events while leaving not-yet-events untouched, biasing the estimate toward the null.Estimation After AdaptationBoth MAMS and RAR produce biased naive treatment effect estimates because the adaptation depends on observed outcomes. This is frequently overlooked in favour of Type I error discussion.The mechanisms differ. In MAMS, an arm surviving to the final analysis has been selected for having performed well at interim, so the naive estimate of its effect is biased upward; investigations stopping early for efficacy are similarly biased, because early stopping occurs disproportionately on random highs. In RAR, the allocation itself depends on accumulating outcomes, which breaks the independence assumptions underlying standard estimators.Regulators care about this because the reported effect size drives labelling and clinical decision-making, not just the yes/no of the hypothesis test. Sponsors should pre-specify bias-adjusted estimation in the SAP: median-unbiased point estimates and confidence intervals based on an appropriate ordering of the sample space are standard for group sequential and MAMS designs. Correction methods for RAR are less standardised, which is itself an argument for simulation-based evaluation of estimator bias and mean squared error at the design stage, not only power.Simulation Requirements and Operating CharacteristicsBefore implementing any adaptive design, extensive simulation studies are required — typically 10,000 or more simulated investigations under multiple scenarios. These simulations must explore the null hypothesis (where all treatments are equivalent), realistic alternative hypotheses with plausible effect sizes, mixed scenarios where some treatments work while others do not, delayed response scenarios, and explicit temporal drift scenarios including operator learning curves. For each simulated investigation, patient outcomes are generated from appropriate probability distributions, adaptation rules are applied, and final sample sizes, allocation ratios, and conclusions are recorded.The resulting operating characteristics are what regulators scrutinise most closely. Under the null, the Type I error rate must remain controlled — conventionally two-sided \alpha = 0.05, or one-sided \alpha = 0.025 for superiority investigations. Under realistic alternatives, the design must achieve adequate power, typically at least 80%, to detect clinically meaningful differences. Adaptive designs typically require a larger maximum sample size than a fixed design while delivering a smaller expected sample size, and both quantities inform operational planning. The allocation distribution should be reported with its variability across simulations, not only its mean, including the proportion of investigations in which more patients are allocated to an inferior arm. The early stopping probability quantifies how often the investigation would terminate for efficacy or futility, and estimator bias and mean squared error should be reported for both naive and bias-adjusted estimates of the primary treatment effect.Sample size determination proceeds iteratively. The design is simulated across a range of total sample sizes under the alternative hypothesis, and the smallest N that achieves the desired power is selected, provided Type I error remains controlled under the null.Sensitivity analyses test robustness to violations of assumptions: varying effect sizes, endpoint timing for RAR, accrual rates, dropout rates, and the magnitude of temporal drift. Simulation results themselves have sampling error. With 10,000 simulations, the Monte Carlo standard error for a 5% Type I error rate is approximately:SE = \sqrt{\frac{0.05 \times 0.95}{10{,}000}} \approx 0.0022giving a 95% confidence interval of roughly 4.6% to 5.4%. For power estimation, use more simulations if the target is close to a regulatory threshold.Where the purpose of the simulation is to choose between MAMS and RAR rather than to characterise a single chosen design, the comparison should be run head-to-head under a common set of scenarios and a common maximum sample size. Reporting each design’s operating characteristics in isolation makes the trade-off invisible.Pre-Specified Simulation Scenario TableThe following scenarios should be pre-specified in the SAP:ScenarioDescriptionPurposeGlobal nullAll treatments are equally effectiveAssess Type I error controlSuperiorityOne treatment is superior to all othersAssess power and expected sample sizeMixedSome treatments work, others do notAssess ability to drop inferior armsDelayed responseOutcomes take time to confirmTest RAR delayed response handlingTemporal driftPopulation or operator skill changes over timeTest confounding of allocation with timeHigh dropoutSignificant patient dropout rateTest robustness to missing dataPre-specifying scenarios prevents “data dredging” by selecting scenarios that make the design look good post-hoc.Regulatory ConsiderationsRegulators globally have become increasingly supportive of adaptive designs, provided they are grounded in mathematical rigour. Support is not uniform across adaptation types, however, and RAR occupies a more contested position than MAMS.Across all regions, regulators require evidence that Type I error remains controlled at the pre-specified significance level under the global null, that the investigation achieves adequate power to detect clinically meaningful differences, and that the design yields efficiency gains when clear winners exist (Pallmann et al., 2018). Regulators increasingly expect simulation code to be submitted for independent verification. Sponsors should ensure simulation code is well-documented and commented, that random number seeds are set for reproducibility, and that the code can be run independently by regulators or their statistical reviewers.United StatesThe FDA’s 2016 guidance Adaptive Designs for Medical Device Clinical Studies is the governing document, with Guidance for the Use of Bayesian Statistics in Medical Device Clinical Trials (CDRH/CBER, 2010) as the relevant companion for Bayesian approaches including RAR. CDRH has historically been more receptive to Bayesian methods than FDA’s drug centres, which is a genuine advantage for device sponsors.The FDA expects adaptation algorithms, whether MAMS stopping boundaries or RAR probability updates, to be detailed in the Statistical Analysis Plan. The SAP must be finalised and locked before any unblinded data are reviewed, ensuring that adaptive features are truly prospective and algorithmic rather than subjective. Pre-specification before the study begins is the regulatory standard. The guidance’s central principle is that modifications are scientifically valid when made without knowledge of outcome results by treatment group, which places the emphasis on firewalls and controlled information flow as much as on the algorithms themselves.European UnionIn the European Union, clinical investigations are governed by the EU Medical Device Regulation (MDR 2017/745) and ISO 14155:2026. Under the MDR, the study protocol is officially designated as the Clinical Investigation Plan (CIP). The CIP must outline the high-level adaptive methodology and justify how the design protects human subjects and minimises risk (Article 62), with statistical design and analysis content specified per ISO 14155:2026 Annex A. The exact statistical algorithms, simulation parameters, and \alpha-spending functions are then specified in a separate, detailed SAP.The SAP should be finalised and locked before the initiation of any adaptive algorithms. While not explicitly required by MDR Article 62 or ISO 14155, this is considered best practice and is expected by the Member State Competent Authorities that authorise clinical investigations.The EMA’s Complex Clinical Trials Questions and Answers (2022) is the current reference point for novel design discussions. Sponsors considering RAR in the EU should engage early and be prepared to demonstrate robust control of temporal drift.United KingdomIn the United Kingdom, device clinical investigations are governed by the UK Medical Devices Regulations 2002 (revised version expected June 2027). The MHRA Innovation Office supports innovative trial designs across both medicines and devices, and MHRA guidance states that it supports complex innovative designs such as umbrella, basket, platform, and master protocol trials, requiring sponsors to justify the design choice, demonstrate that each adaptation and the investigation as a whole are safe and scientifically sound, and describe how the integrity of results will be maintained.Sponsors should note a jurisdictional distinction that is easy to miss: much of the MHRA’s published adaptive-design engagement sits in the CTIMP (medicines) space. The methodological expectations transfer readily; the regulatory route does not. Early engagement with the MHRA Innovation Office is the appropriate mechanism for a novel device design.Pre-Submission EngagementAcross all regions, pre-submission interactions (FDA Pre-Subs, MHRA Innovation Office or scientific advice meetings, or Expert Consultations with EU Competent Authorities) should include the proposed CIP and draft SAP, simulation results demonstrating operating characteristics under both stationary and drift scenarios, clear rationale for methodological choices, the firewall and information-flow plan, and detailed plans for Independent Data Monitoring Committee (IDMC) involvement.Implementation RequirementsImplementing adaptive randomisation requires careful consideration of operational constraints. MAMS designs require \alpha-spending function calculations at discrete intervals, which can be performed on periodic clean data cuts.RAR designs demand considerably more: real-time or group-sequential Bayesian updating, Monte Carlo sampling for Thompson sampling, and automated allocation probability updates. The system architecture flows from data entry through real-time databases to statistical engines that feed the randomisation system. Every link in that chain becomes a potential single point of failure that can silently corrupt allocation. RAR investigations inherently require greater statistical analysis plan complexity, increased programming effort for the creation and validation of simulation and allocation code, and heightened data management complexity. The randomisation system requires formal validation, and the audit trail must be sufficient to reconstruct why each allocation probability took the value it did at the time it was applied.A further consideration is specific to devices: most device investigations cannot be fully blinded, which interacts badly with RAR. A visibly skewed allocation ratio leaks interim results to investigators and patients, potentially introducing performance bias and influencing which patients are approached, how they are consented, and how subjective endpoints are assessed. Maintaining allocation concealment is materially more challenging with RAR than with fixed randomisation or MAMS. The FDA’s 2016 guidance emphasises the importance of firewalls to prevent operational bias — measures that deserve more than a single passing IDMC mention in the protocol.Decision Framework for Method SelectionThe framework below should be read alongside the comparison table in the Fundamental Differences section.Choose Multi-Arm Multi-Stage (MAMS) designs when:There are well-defined interim analysis timepoints (e.g., safety run-ins, planned efficacy looks)Primary endpoints require substantial follow-up timeThe investigation features multiple treatment arms where dropping inferior arms is preferred over skewing allocationOperator learning curves or other temporal trends are expectedThe investigation cannot be blindedThere is a regulatory preference for pre-specified, discrete adaptation rules and fixed randomisation between looksReal-time data processing capabilities are limitedChoose Response-Adaptive Randomisation (RAR) when:Rapid endpoint assessment is possible (minutes to days), or group-sequential updating is operationally feasibleThere is a strong ethical imperative to minimise exposure to inferior treatments without dropping them entirelyThe patient population is homogeneous and stable over the enrolment periodRobust, real-time data systems are availableThe investigation prioritises within-investigation patient benefit, and the associated power cost has been quantified and acceptedAdequate mitigations against temporal drift are pre-specified: burn-in, control-arm protection, and drift-adjusted analysisConsider hybrid approaches when:Different endpoints have different assessment timelinesBoth early safety signals (MAMS) and longer-term allocation efficiency (RAR) matterRegulatory discussions suggest openness to novel designsWhile MAMS and RAR are often presented as alternatives, in practice they exist on a spectrum. I-SPY 2 and REMAP-CAP combine response-adaptive randomisation with arm graduation and dropping; STAMPEDE uses a pure MAMS framework. The choice is not binary but should reflect the specific operational and scientific context of the investigation.RAR’s patient-level adaptation offers genuine ethical advantages but demands flawless data systems, a stable population, and acceptance of a power and estimation cost. MAMS provides more operational control and traditional regulatory familiarity but may miss opportunities for real-time optimisation. Both approaches require extensive simulation studies to demonstrate operating characteristics under realistic, including adverse, scenarios. The choice between them should be driven by which method best serves the specific combination of scientific questions, operational constraints, and regulatory pathway.ReferencesBai, Z. D., Hu, F., & Rosenberger, W. F. (2002). Asymptotic properties of adaptive designs for clinical trials with delayed response. The Annals of Statistics, 30(1), 122–139.Berry, S. M., Carlin, B. P., Lee, J. J., & Müller, P. (2010). Bayesian Adaptive Methods for Clinical Trials. Chapman and Hall/CRC.European Medicines Agency. (2022). Complex Clinical Trials — Questions and Answers.International Organization for Standardization (ISO). (2021). ISO 14155:2026 — Clinical investigation of medical devices for human subjects — Good clinical practice.International Council for Harmonisation (ICH). (2025). ICH E20 Guideline on Adaptive Designs for Clinical Trials. Step 2b draft.Magirr, D., Jaki, T., & Whitehead, J. (2012). A generalized Dunnett test for multi-arm multi-stage clinical studies with treatment selection. Biometrika, 99(2), 494–501.Pallmann, P., Bedding, A. W., Choodari-Oskooei, B., et al. (2018). Adaptive designs in clinical trials: why use them, and how to run and report them. BMC Medicine, 16, Article 29.Proschan, M. A., & Evans, S. (2020). Resist the temptation of response-adaptive randomization. Clinical Infectious Diseases, 71(11), 3002–3004.Robertson, D. S., Lee, K. M., López-Kolkovska, B. C., & Villar, S. S. (2023). Response-adaptive randomization in clinical trials: from myths to practical considerations. Statistical Science, 38(2), 185–208.Saville, B. R., Berry, D. A., Berry, N. S., Viele, K., & Berry, S. M. (2022). The Bayesian time machine: accounting for temporal drift in multi-arm platform trials. Clinical Trials, 19(5), 490–501.Thall, P. F., Fox, P. S., & Wathen, J. K. (2015). Statistical controversies in clinical research: scientific and ethical problems with adaptive randomization in comparative clinical trials. Annals of Oncology, 26(8), 1621–1628.U.S. Food and Drug Administration. (2010). Guidance for the Use of Bayesian Statistics in Medical Device Clinical Trials: Guidance for Industry and FDA Staff. CDRH/CBER.U.S. Food and Drug Administration. (2016). Adaptive Designs for Medical Device Clinical Studies: Guidance for Industry and Food and Drug Administration Staff.Wason, J. M. S., & Trippa, L. (2014). A comparison of Bayesian adaptive randomization and multi-stage designs for multi-arm clinical trials. Statistics in Medicine, 33(13), 2206–2221.Wei, L. J., & Durham, S. (1978). The randomized play-the-winner rule in medical trials. Journal of the American Statistical Association, 73(364), 840–843.Wilson, I., Julious, S., Yap, C., Todd, S., & Dimairo, M. (2025). Response adaptive randomisation in clinical trials: current practice, gaps and future directions. Statistical Methods in Medical Research.Zhai, G., Li, Y., Zhang, L., & Hu, F. (2024). Group response-adaptive randomization with delayed and missing responses. Statistics in Medicine, 43, 5047–5059.