Sample Size Justification in the Era of MDR: Navigating Power, Tolerance, and Assurance

Series: Advanced Biostatistics for MedTech: Bridging Clinical Evaluation and Engineering

The Regulatory Rejection of “Industry Standard” Sample Sizes

One of the most frequently cited Major Non-Conformities in Notified Body audit reports under the EU Medical Device Regulation (MDR 2017/745) and IVDR 2017/746 is the inadequate statistical justification of sample sizes. During design control reviews and Clinical Evaluation Report (CER) appraisals, technical reviewers consistently reject sample size rationale predicated on “historical precedent” or “industry standard practice.”

Justifying sample sizes for medical devices requires a fundamentally different statistical framework to pharmaceutical studies. The objective is not always to evaluate a clinical treatment effect, but to demonstrate physical reliability, conformance to engineering specifications, or an acceptable probability of use error. A pharmaceutical dossier is anchored in clinical efficacy data, whereas a medical device technical documentation file is often heavily dominated by non-clinical Verification and Validation (V&V) data. This is especially true for line extensions or devices relying on substantial equivalence (510(k)) and equivalent device routes (MDR Article 61). In these cases V&V activities constitute the primary evidence of safety and performance, often without the need for a new clinical trial. V & V activities can consit of bench testing, biocompatibility, usability engineering (IEC 62366-1), and manufacturing validation. Even for entirely novel devices requiring pivotal clinical investigations, thorough V&V is a prerequisite gatekeeper.

As Pardo (2023) demonstrates, applying standard power analyses to these V&V activities is a category error. The statistical objectives of device validation are fundamentally different from those of clinical trials. To construct a defensible sample size justification for a technical documentation file, the biostatistician must navigate a complex interplay between confidence, power, tolerance intervals, and reliability assurance.

The Triad of Statistical Intervals: Confidence, Prediction, and Tolerance

The root cause of statistical friction in device V&V is the conflation of three distinct statistical intervals. Graduate biostatistics programmes extensively cover Confidence Intervals (CIs), which characterise the uncertainty around a population parameter (e.g., the mean). However, V&V activities frequently require Prediction Intervals or Tolerance Intervals.

Pardo (2023) provides a dissection of Neyman’s confidence framework, emphasising that a 95% CI constructed from a sample of size $n$ implies that if the experiment were repeated infinitely, 95% of the resulting intervals would contain the true population mean. A CI says nothing about the proportion of individual devices that will meet specification.

A Prediction Interval addresses the next single observation. A 95% prediction interval provides bounds within which the next single randomly sampled unit will fall with 95% probability.

A Tolerance Interval, governed by ISO 16269-6, provides bounds that contain a specified proportion ( p ) of the population with a specified confidence level ( \gamma ). When validating a physical dimension against a specification limit, a Notified Body expects evidence that virtually all individual units will conform, not just that the average unit conforms.

For a normally distributed characteristic with unknown mean and variance, a one-sided tolerance bound is derived exactly using the non-central t-distribution. A two-sided tolerance interval has no exact closed form and is typically computed using a chi-square approximation (such as the Howe method), which is the basis for the tabulated factors in ISO 16269-6. For a two-sided interval, the bounds are constructed as:

\bar{X} \pm k S

Where \bar{X} is the sample mean and S is the sample standard deviation.

When a manufacturer states that a sample size of n=30 was chosen to “guarantee 95% confidence,” this is mathematically meaningless without specifying what is being guaranteed. If the objective is to demonstrate that 95% of the population falls within the specification limits with 95% confidence (a 95/95 tolerance interval), the required sample size and the resulting interval width are substantially different from a 95% confidence interval for the mean. Biostatisticians must enforce this strict terminology in V&V protocols.

The c=0 Acceptance Sampling Logic

For attribute data (pass/fail, go/no-go), sample size justification cannot rely on standard power curves. Instead, it relies on the operating characteristics of acceptance sampling plans. Under ISO 2859-1 and the Squeglia Zero Acceptance Number (c=0) framework, sample sizes are derived from the binomial probability distribution.

In a c=0 plan, if a single defective unit is found in a sample of size n, the lot or batch fails the validation. The statistical rationale is framed around two parameters:

1. P: The desired probability of passing the test (e.g., 95%).
2. p_0: The maximum acceptable defective rate in the population. In this confidence framing,
3. p0​ represents the consumer’s-risk point, often termed the Rejectable Quality Limit (RQL), Limiting Quality (LQ).

Under a binomial model, the probability of accepting a lot (finding zero defectives in a sample of size n ) when the true population defective rate is p_0 is:

If the objective is to be P confident that the lot has a defective rate no higher than p_0, the required sample size is derived algebraically:

For example, to achieve 95% confidence (P = 0.95) that the true defective rate is less than 5% (p_0 = 0.05), the required sample size is n = 59. This algebraic derivation is the absolute minimum defence required in a V&V protocol.

The binomial derivation assumes an infinite population. For finite populations (e.g., a specific manufacturing batch of 500 units), the hypergeometric distribution is technically more accurate, though the binomial is often used as a convenient large‑lot approximation; for small lots the exact hypergeometric calculation should be used. The c=0 logic is entirely frequentist. It does not account for prior knowledge of the manufacturing process. For post-market surveillance or legacy device validations where historical batch data exists, Bayesian augmentation of the c=0 plan can significantly reduce the required sample size while still providing a high level of assurance, provided the prior is well justified and sensitivity analyses confirm the conclusions are not unduly driven by optimistic assumptions.

Variables Sampling Plans and the Non-Central t-Distribution

For continuous data (e.g., tensile strength, flow rate, assay precision), attribute sampling is statistically inefficient. Instead, Variables Sampling Plans based on the Process Capability Index (Cpk) are mandated.

The hypothesis test is structured to demonstrate that the true population Cpk exceeds a target value K_0 (e.g., K_0 = 1.0 , corresponding to 3\sigma capability or 99.73% conformance):

To demonstrate this with 95% confidence, a Notified Body expects to see a lower confidence bound on Cpk. This is where many engineering validations fail statistically. A common, yet fatal, flaw is to set the acceptance criterion at the target value (e.g., requiring the sample Cpk to be \geq 1.0).

If the true population Cpk is exactly 1.0, the probability that the sample Cpk will exceed 1.0 is only approximately 52.7%. Setting the bar at 1.0 provides essentially no regulatory assurance (roughly 50% confidence) that the true Cpk is actually 1.0.

To demonstrate with 95% confidence that the true Cpk is at least 1.0, the acceptance criterion must be raised. The sampling distribution of the sample Cpk is related to the non-central t-distribution. The non-centrality parameter (\delta_{ncp}) is a function of the sample size and the target capability:

\delta_{ncp} = 3 \sqrt{n} K_0

For a sample size of n = 36 and a target K_0 = 1.0, the math dictates that the sample Cpk must be approximately 1.27 to assert with 95% confidence that the true population Cpk is at least 1.0. Under this stringent criterion, if the true Cpk is exactly 1.0, the probability of passing the test is only 5% (the \alpha risk). This correctly places the statistical protection on the regulator/consumer, ensuring that a marginally capable process will not easily pass validation.

These critical values must be derived using the non-central t-distribution. This power curve plots the probability of passing the test against various true population Cpk values. It’s inclusion in the SAP is the gold standard for defending continuous data sample sizes to a Notified Body.

The Concept of Assurance (Bayesian Expected Power)

A persistent flaw in frequentist sample size justification is the assumption that the true effect size (or variance, or defective rate) is known with certainty. In reality, the true parameters are estimates derived from small pilot studies, engineering tolerances, or literature reviews. If the true effect size is overestimated during planning, the study will be underpowered. If it is underestimated, the study is overpowered, wasting resources and potentially exposing subjects to unnecessary risk.

While MDR Article 61 requires robust clinical evaluation that accounts for uncertainty, it does not mandate a specific statistical approach. Relying solely on standard frequentist power calculations, which assume the true effect size is known with certainty, can be a limitation. The advanced statistical solution to this limitation is the concept of Assurance (also known as Bayesian Expected Power or Probability of Success).

Assurance integrates the uncertainty of the true parameter over its prior distribution. Instead of calculating power at a single point estimate, assurance calculates the expected power over the entire space of plausible parameter values.

Let \theta be the true parameter of interest (e.g., the true mean difference, the true variance, or the true defective rate). Frequentist power is conditional on a specific value of \theta:

Assurance, however, marginalises over the prior distribution \pi(\theta):

For medical devices, deriving the prior distribution \pi(\theta) is highly tractable. Device development is iterative; bench-testing data, computer simulations, and animal studies provide robust prior information regarding device performance. By formally encoding this prior data into a distribution (e.g., a conjugate Beta distribution for a binomial defective rate, or a Normal-Inverse-Gamma for normal means and variances), the assurance calculation provides a much more realistic estimate of the probability that the V&V activity will succeed.

The strongest regulatory precedent for Bayesian methods in the medical device industry comes from the FDA/CDRH’s 2010 Guidance for the Use of Bayesian Statistics in Medical Device Clinical Trials, though EU Notified Bodies are increasingly receptive to these approaches when they transparently incorporate the totality of evidence rather than relying on arbitrary point estimates. If assurance is used, the SAP must detail the elicitation of the prior distribution and conduct sensitivity analyses to demonstrate that the calculation is not unduly influenced by overly optimistic prior assumptions.

Reverse-Engineering Sample Sizes: The Engineering Constraint

In the medtech industry, the biostatistician frequently encounters a scenario where the sample size is dictated by physical, ethical, or economic constraints rather than statistical optimisation. For example:

• A wear simulator may only have 6 test stations.
• Only 10 animal subjects are ethically justifiable for a chronic toxicity study.
• The budget allows for the destruction of only 15 units for fatigue testing.

When the sample size nis fixed by external constraints, the statistical paradigm shifts from “What n is required to achieve 95% power?” to “What statistical assurance can be defensibly claimed with this fixed n?”

This requires reverse-engineering the statistical parameters. Using the c=0 binomial logic, if n=10 is the absolute maximum, and 95% confidence (P=0.95) is required, the maximum defective rate p_0 that can be claimed is derived by rearranging the binomial formula:

For n=10 and P=0.95, the claimable p_0 is approximately 0.258 (25.8%). If the engineering team requires a claim of 5% defective rate, n=10 is demonstrably insufficient. The biostatistician must clearly communicate this gap between engineering ambition and statistical reality.

For continuous data with a fixed n, reverse-engineering requires calculating the precision of the estimate. If n=15, the width of the 95/95 tolerance interval will be wide. The biostatistician must calculate the expected width and determine if the tolerance interval will realistically fit within the engineering specification limits. If the specification limits are narrow, n=15 will yield a tolerance interval that exceeds the limits, resulting in a validation failure.

Presenting a “Reverse-Engineered Assurance Analysis” in the technical documentation file demonstrates to the Notified Body that the manufacturer understands the statistical limitations of their study design. It shifts the regulatory conversation from “Why did you choose this sample size?” to “Given this sample size, here is the maximum statistical risk we are assuming,” which is then formally documented in the Risk Management File (ISO 14971:2019).

Usability Validation and the Probability of Use Error

A specific area where sample size justification routinely fails regulatory scrutiny is in summative usability validation (IEC 62366-1:2015). The objective of usability validation is to demonstrate that the device can be used safely and effectively by the intended users without serious use errors.

The FDA guidance on human factors engineering and the EU MDR requirements for usability engineering dictate that usability testing must sample a representative population of users. However, the statistical defence of this sample size is uniquely challenging because use errors are ideally rare events.

If a critical use error has an acceptable upper bound of 1 in 10,000 occurrences, demonstrating this via a standard c=0 attribute sampling plan would require an astronomical sample size:

Testing 30,000 participants in a usability study is generally impractical. Therefore, the statistical justification for usability testing cannot rely on frequentist attribute sampling alone. Under FDA human-factors guidance and IEC 62366-1, summative usability validation is treated as a primarily qualitative exercise. This entails surfacing use errors and analysing their root causes with roughly15 participants per user group serving as a problem-discovery heuristic, rather than a statistically powered estimate of an error rate. To strengthen the regulatory defence, a biostatistician may supplement this qualitative validation with a Bayesian hierarchical model that combines the summative test data with prior data from formative studies, heuristic evaluations, and predicate device post-market surveillance. By establishing a prior distribution for the use error rate, the posterior distribution can be evaluated to provide additional quantitative assurance that the upper 95% credible bound for the critical use error rate falls below the acceptable threshold.

This approach aligns with the MDR’s emphasis on continuous benefit-risk evaluation and the integration of post-market data into the clinical evaluation cycle.

Documenting the Defence

A robust sample size justification section in a V&V protocol or CER must contain the following elements to survive Notified Body scrutiny:

1. Objective Statement: Explicitly state whether the objective is to estimate a parameter (requiring a CI), bound a future observation (requiring a prediction interval), or bound a proportion of the population (requiring a tolerance interval).
2. Statistical Framework: Specify the exact formula or distribution used. If using a c=0 plan, provide the binomial derivation. If using a Cpk plan, specify the non-central t-distribution parameters.
3. Assumptions: Detail the assumed variance, the assumed defective rate, or the prior distribution. Provide the rationale for these assumptions based on pilot data or engineering analysis.
4. Risk Integration: Map the statistical parameters (e.g., \alpha, \beta, p_0) directly to the risk acceptance criteria in the Risk Management File. If a hazard is classified as “Catastrophic,” the statistical assurance must be correspondingly high (e.g., 99% confidence).
5. Sensitivity Analysis: If Bayesian assurance is used, demonstrate the impact of varying the prior distribution. If frequentist power is used, demonstrate the impact of varying the assumed effect size.

Conclusion

Sample size justification in the medical device industry is a multi-faceted statistical defence that encompasses engineering constraints, clinical risk, and regulatory expectation. By use of acceptance sampling, the derivation of non-central t-distributions for Cpk, and the Bayesian integration of prior data via assurance calculations, a biostatistician can transform the sample size justification from a regulatory vulnerability into a quantitative asset.

References:

1. Pardo, S. A. (2023). Statistical Methods and Analyses for Medical Devices. Springer.
2. European Parliament and Council. (2017). Regulation (EU) 2017/745 on medical devices (MDR), Articles 61, 62, 83, and Annex XV. Official Journal of the European Union.
3. International Organization for Standardization. (1999). ISO 2859-1:1999 – Sampling procedures for inspection by attributes.
4. Squeglia, N. L. (2008). Zero Acceptance Number Sampling Plans (4th ed.). ASQ Quality Press.
5. International Organization for Standardization. (2014). ISO 16269-6:2014 – Statistical interpretation of data – Part 6: Determination of statistical tolerance intervals.
6. International Electrotechnical Commission. (2015). IEC 62366-1:2015 – Medical devices – Application of usability engineering to medical devices.
7. US Food and Drug Administration. (2016). Applying Human Factors and Usability Engineering to Medical Devices.
8. US Food and Drug Administration. (2010). Guidance for the Use of Bayesian Statistics in Medical Device Clinical Trials.