Quick Guide: Simulation Evidence for Frequentist Adaptive Designs in Medtech Clinical Trials

A fixed frequentist design, such as a standard two-arm randomised superiority study, generally does not require simulation. Its statistical properties come from established formulas. Closed-form or table-based power calculations tell you the sample size, and the Type I error rate follows from the test you specified. Simulation enters the picture when the study design gets messy in ways the standard formulas can’t absorb, for instance complex survival endpoints, clustered data, very small and/or heterogeneous samples.

An adaptive frequentist design is a different matter. The moment you add response-adaptive randomisation, interim analyses with stopping rules, or the dropping of inferior arms, the standard assumptions break down. How much data you have at the interim, how many patients are still awaiting outcomes, which arms are still running, and how randomisation has shifted are all now random quantities whose behaviour depends on everything else in the study design. Pure group-sequential stopping still has well-developed functions(alpha spending functions), but a realistic assembled design (adaptive randomisation plus arm dropping plus delayed outcomes plus patient dropout) has no analytic route. If you want to know its Type I error rate, its power, or its expected sample size, simulation is effectively the only practical option.

Adaptive designs are increasingly attractive in medtech. They allow clinical investigations that are smaller when one device clearly outperforms, that stop exposing patients to inferior treatment sooner, and that give more patients the better option while the investigation is still running. But that flexibility has a price. For most adaptive designs, the statistical properties regulators care about (Type I error, power, expected sample size) cannot be derived analytically. They have to be demonstrated by simulation, before the investigation begins, across the full range of scenarios the design might encounter. Before implementing any adaptive design, extensive simulation studies are required, typically 10,000 or more simulated investigations under different scenarios.

Simulations of frequentist adaptive designs must test:

• The null hypothesis (where all treatments are equivalent) to assess Type I error control
• Realistic alternative hypotheses with plausible effect sizes to assess power
• Mixed scenarios (where some treatments work while others do not) to assess the ability to drop inferior arms
• Delayed response scenarios to test RAR’s handling of data that takes time to confirm
• High dropout scenarios to assess robustness to missing data

For each simulated investigation, patient outcomes are generated from appropriate probability distributions, adaptation rules are applied, and final sample sizes, allocation ratios, and conclusions are recorded.

Pre-Specified Simulation Scenario Table for the SAP

The following scenarios should be pre-specified in the SAP:

Pre-specifying scenarios prevents “data dredging” by selecting scenarios that make the design look good post-hoc. It is better to show regulators the design’s full range rather than cherry-picking it’s best day.

Simulations of frequentist adaptive designs should report the following key operating characteristics:

• Type I error rate: The proportion of simulations where the null hypothesis is falsely rejected (should be ≤ 0.05 under the global null scenario)
• Power: The proportion of simulations where the true treatment effect is correctly detected (typically ≥ 80% for realistic effect sizes)
• Expected sample size: The average total sample size required across simulations, which informs operational planning
• Allocation distribution: The proportion of patients assigned to each treatment arm at the end of the investigation
• Early stopping probability: The proportion of simulations that stop early for efficacy, futility, or at each interim analysis

Sample Size Simulation for frequentist adaptive designs

Sample size determination for adaptive designs typically proceeds as follows:

1. Define the clinically meaningful effect size (e.g., 15% reduction in target vessel revascularisation)
2. Specify the desired power (e.g., 80% or 90%)
3. For a range of total sample sizes (e.g., N = 100, 150, 200, 250, 300), simulate the design under the alternative hypothesis
4. Identify the smallest N that achieves the desired power
5. Verify that Type I error remains controlled at N under the null hypothesis

Important: Sample size requirements for adaptive designs are often larger than for fixed designs when no treatment is effective, but smaller when a clear winner emerges. This trade-off should be explicitly evaluated in simulations and communicated to stakeholders, particularly anyone planning the budget and timeline.

Sensitivity analysis of frequentist adaptive design simulations

It should be assumed that assumptions will be violated. Sensitivity analyses should test the design’s robustness by varying:

• Varying the true treatment effect: Simulate under optimistic, realistic, and pessimistic effect sizes
• Varying endpoint timing: Test the impact of delayed responses on RAR performance
• Varying accrual rates: Test whether slow enrolment affects the design’s operating characteristics
• Varying dropout rates: Test the impact of patient dropout on the adaptation algorithm

If the design’s performance degrades sharply under one of these variations, it is better to discover that in silico than when the clinical investigation is already well underway.

How many simulations are enough?

Simulation results carry sampling error. With 10,000 simulations, the Monte Carlo standard error for estimating a 5% Type I error rate is approximately:$$SE=\sqrt{\frac{0.05\times 0.95}{10,000}}\approx 0.0022$$SE=10,0000.05×0.95​​≈0.0022

This means the 95% confidence interval for the true Type I error rate is approximately 4.6% to 5.4%. That is fine if the design’s true Type I error rate sits well below 0.05, but uncomfortably wide if it sits near the boundary. For power estimates close to a regulatory threshold, run more simulations. The cost is only compute time.

Regulatory considerations for frequentist simulation evidence

Regulators require evidence that:

• Type I error remains strictly controlled (below 0.05) under the null hypothesis
• The investigation achieves adequate power (typically ≥ 80%) to detect clinically meaningful differences
• The design yields efficiency gains when clear winners exist (Pallmann et al., 2018)

Regulators increasingly expect simulation code to be submitted for independent verification. The MHRA, in particular, often requests that sponsors provide the simulation code used to derive operating characteristics. Sponsors should ensure that:

• The code can be run independently by regulators or their statistical reviewers
• The code is well-documented and commented
• Random number seeds are set for reproducibility

Regulators globally have become increasingly supportive of adaptive designs. The FDA and EMA have both published guidance signalling openness to well-justified adaptations, and the same thinking has been extended to medical devices. The consistent message across these documents is that adaptation is welcome when it is pre-specified, its statistical properties are well characterised, and operational biases are controlled. Simulation is the evidence that ties all three together, and it is increasingly the artefact regulators will want to inspect for themselves.

The bottom line

• There is no analytic shortcut. For adaptive designs, operating characteristics come from simulation or not at all. Budget the statistical expertise and time accordingly.
• 10,000 simulations per scenario is a floor, not a ceiling. If estimates sit near a threshold (5% Type I error, 80% power), increase the replications.
• Pre-specify everything in the SAP, including the scenario table, so simulation evidence is part of the plan rather than an afterthought.
• Simulate the design’s bad days: the global null, mixed effects, delayed responses, high dropout, and slow accrual.
• Write simulation code as if a regulatory statistical reviewer will run it. If the MHRA requests it, they will.

For a look at the Bayesian approach to adaptive designs in medtech, see here.

References:

Pallmann P, Bedding AW, Choodari-Oskooei B, et al. Adaptive designs in clinical trials: why use them, and how to run and report them. BMC Medicine. 2018;16:29.