Sample Size Justification in the Era of MDR: Navigating Power, Tolerance, and Assurance

Series: Advanced Biostatistics for MedTech: Bridging Clinical Evaluation and Engineering

The Regulatory Rejection of “Industry Standard” Sample Size

A frequently cited Major Non-Conformity in Notified Body audit reports under the EU Medical Device Regulation (MDR 2017/745) and IVDR 2017/746 is the inadequate statistical justification of sample size estimates. During design control reviews and Clinical Evaluation Report (CER) appraisals, technical reviewers consistently reject sample size rationale predicated on “historical precedent” or “industry standard practice.”

Justifying sample size estimates for medical device studies requires a different statistical framework to pharmaceutical studies. The goal of medical device studies is not always to evaluate a clinical treatment effect. Other common objectives include demonstrating physical reliability, conformance to engineering specifications, or an acceptable probability of use error. A medical device technical documentation file can be heavily dominated by non-clinical Verification and Validation (V&V) data. This is especially true for line extensions or devices relying on substantial equivalence (510(k)) and equivalent device routes (MDR Article 61). In these cases V&V activities constitute the primary evidence of safety and performance, often without the need for a new clinical trial. V&V activities can include bench testing, biocompatibility, usability engineering (IEC 62366-1), and manufacturing validation. Even for entirely novel devices requiring pivotal clinical investigations, thorough V&V is a prerequisite.

In sample size estimation for conformance V&V, standard power calculations for detecting a difference in means are replaced by power calculations to assess a limit built around interval estimates. To construct a valid sample size estimation for a technical documentation file, the biostatistician must navigate between confidence, tolerance intervals, reliability assurance, and the probability of validation success (Pardo, 2023).

The Triad of Statistical Intervals: Confidence, Prediction, and Tolerance

A root cause of statistical friction in device V&V can be the conflation of three distinct statistical intervals. Confidence Intervals (CIs) characterise the uncertainty around a population parameter (e.g., the mean). A 95% CI constructed from a sample of size n implies that if the experiment were repeated infinitely, 95% of the resulting intervals would contain the true population mean. A CI says nothing about the proportion of individual devices that will meet specification. V&V activities frequently require Prediction Intervals and/or Tolerance Intervals instead.

A Prediction Interval addresses the next single observation. A 95% prediction interval provides bounds within which the next single randomly sampled unit will fall with 95% probability.

A Tolerance Interval, governed by ISO 16269-6, provides bounds that contain a specified proportion ( p ) of the population with a specified confidence level ( \gamma ). When validating a physical dimension against a specification limit, a Notified Body expects evidence that virtually all individual units will conform, not just that the average unit conforms.

For a normally distributed characteristic with unknown mean and variance, a one-sided tolerance bound is derived exactly using the non-central t-distribution. A two-sided tolerance interval has no exact closed form and is typically computed using a chi-square approximation (such as the Howe method), which is the basis for the tabulated factors in ISO 16269-6. For a two-sided interval, the bounds are constructed as:

\bar{X} \pm k S

Where \bar{X} is the sample mean and S is the sample standard deviation.

To state that a sample size of n=30 was chosen to “guarantee 95% confidence,” is mathematically meaningless without specifying what is being guaranteed. If the goal is to demonstrate that 95% of the population falls within the specification limits with 95% confidence (a 95/95 tolerance interval), the required sample size and the resulting interval width are substantially different from a 95% confidence interval for the mean. Biostatisticians must enforce this terminology distinction in V&V protocols.

The c=0 Acceptance Sampling Logic

For attribute data (pass/fail, go/no-go), sample size justification cannot rely on standard power curves. Instead, it relies on the operating characteristics of acceptance sampling plans. Under ISO 2859-1 and the Squeglia Zero Acceptance Number (c=0) framework, sample sizes are derived from the binomial probability distribution.

In a c=0 plan, if a single defective unit is found in a sample of size n, the lot or batch fails the validation. The statistical rationale is framed around two parameters:

1. P: The desired confidence level (e.g., 0.95) that the true defective rate is ≤ p₀.
2. p_0: The maximum acceptable defective rate in the population. In this confidence framing,
3. p0​ represents the consumer’s-risk point, often termed the Rejectable Quality Limit (RQL), Limiting Quality (LQ).

Under a binomial model, the probability of accepting a lot (finding zero defectives in a sample of size n ) when the true population defective rate is p_0 is:

If the objective is to be P confident that the lot has a defective rate no higher than p_0, the required sample size is derived algebraically:

For example, to achieve 95% confidence (P = 0.95) that the true defective rate is less than 5% (p_0 = 0.05), the required sample size is n = 59. While p_0 is the consumer’s-risk point, it coincides with the exact one-sided 95% upper confidence bound (e.g., 0 failures in 59 units yields an exact upper bound of 4.951%). This derivation is the minimum justification required in a V&V protocol.

The binomial derivation assumes an infinite population and is often used as a convenient large‑lot approximation. For finite populations (e.g., a specific manufacturing batch of 500 units), the hypergeometric distribution is technically more accurate. For small lots the exact hypergeometric calculation should be used. The c=0 logic is frequentist. It does not account for prior knowledge of the manufacturing process. For post-market surveillance or legacy medical device validations where historical batch data exists, Bayesian augmentation of the c=0 plan can significantly reduce the required sample size while still providing a high level of assurance. The prior must be well justified and sensitivity analyses used to confirm that conclusions are not unduly driven by optimistic assumptions.

Variables Sampling Plans and the Non-Central t-Distribution

For continuous data (e.g., tensile strength, flow rate, assay precision), attribute sampling is statistically inefficient. Instead, Variables Sampling Plans based on the Process Capability Index (Cpk) are mandated.

The hypothesis test is structured to demonstrate that the true population Cpk exceeds a target value K_0 (e.g., K_0 = 1.0 , corresponding to 3\sigma capability or 99.73% conformance):

To demonstrate with 95% confidence that the true Cpk is at least 1.0, a Notified Body expects the lower 95% confidence bound of the sample Cpk to exceed 1.0. A common, yet fatal, flaw is to simply set the acceptance criterion at the target value (e.g., requiring the sample Cpk to be \geq 1.0). If the true population Cpk is exactly 1.0, the probability that the sample Cpk will exceed 1.0 is approximately 52.7%. Setting the bar at 1.0 provides essentially no regulatory assurance that the true Cpk is actually 1.0.

To be sure the lower 95% confidence bound exceeds 1.0, the acceptance criterion must be raised. For a sample size of n = 36 and a target K_0 = 1.0, the required sample Cpk must be approximately 1.27. Under this stringent criterion, if the true Cpk is exactly 1.0, the probability of the sample passing the test is only 5% (the \alpha risk). This correctly places the statistical protection on the regulator/consumer side, so that a marginally capable process will not easily pass.

These critical values are derived using the non-central t-distribution, where the non-centrality parameter (\delta_{ncp}) is a function of the sample size and the target capability:

\delta_{ncp} = 3 \sqrt{n} K_0

Plotting the probability of passing against a range of true population Cpk values produces the operating characteristic curve for the plan. Its inclusion in the SAP is the gold standard for defending continuous data sample sizes to a Notified Body.

The Concept of Assurance (Bayesian Expected Power)

A persistent flaw in frequentist sample size justification is the assumption that the true effect size (or variance, or defective rate) is known with certainty. In reality, the true parameters are estimates derived from small pilot studies, engineering tolerances, or literature reviews. If the true effect size is overestimated during planning, the study will be underpowered. If it is underestimated, the study is overpowered, wasting resources and potentially exposing subjects to unnecessary risk.

The MDR Article 61 requires robust clinical evaluation that accounts for uncertainty but it doesn’t mandate a specific statistical approach. Relying solely on standard frequentist power calculations, which assume the true effect size is known with certainty, is not ideal. The proper statistical approach involves the concept of Assurance (also known as Bayesian Expected Power or Probability of Success).

Assurance integrates the uncertainty of the true parameter over its prior distribution. Instead of calculating power at a single point estimate, assurance calculates the expected power over the entire space of plausible parameter values.

Let \theta be the true parameter of interest (e.g., the true mean difference, the true variance, or the true defective rate). Frequentist power is conditional on a specific value of \theta:

Assurance marginalises over the prior distribution \pi(\theta):

For medical devices, deriving the prior distribution \pi(\theta) is highly tractable. Device development is iterative; bench-testing data, computer simulations, and animal studies provide robust prior information regarding device performance. By formally translating this prior data into a distribution (e.g., a conjugate Beta distribution for a binomial defective rate, or a Normal-Inverse-Gamma for normal means and variances), the assurance calculation more realistically estimates the probability that the V&V activity will succeed.

The strongest regulatory precedent for Bayesian methods in the medical device industry comes from the FDA/CDRH’s 2010 Guidance for the Use of Bayesian Statistics in Medical Device Clinical Trials. EU Notified Bodies are increasingly receptive to these approaches when they transparently incorporate the totality of evidence rather than relying on arbitrary point estimates. If assurance is used, the SAP must detail the elicitation of the prior distribution and conduct sensitivity analyses to demonstrate that the calculation is not unduly influenced by overly optimistic prior assumptions.

Reverse-Engineering Sample Sizes: The Engineering Constraint

In the medtech industry, sample size is commonly dictated by physical, ethical, or economic constraints. For example:

• A wear simulator may only have 6 test stations.
• Only 10 animal subjects are ethically justifiable for a chronic toxicity study.
• The budget allows for the destruction of only 15 units for fatigue testing.

When the sample size nis fixed by external constraints, the statistical paradigm shifts from “What n is required to achieve 95% power?” to “What statistical assurance is valid with this fixed n?”

This requires reverse-engineering of the statistical parameters. Using the c=0 binomial logic, if n=10 is the absolute maximum, and 95% confidence (P=0.95) is required, the maximum defective rate p_0 that can be claimed is derived by rearranging the binomial formula:

For n=10 and P=0.95, the claimable p_0 is approximately 0.259 (25.9%). If the engineering team requires a claim of 5% defective rate, n=10 is demonstrably insufficient. The biostatistician must clearly communicate this gap between engineering ambition and statistical reality.

For continuous data with a fixed n, reverse-engineering requires calculating the precision of the estimate. If n=15, the width of the 95/95 tolerance interval will be wide. The biostatistician must calculate the expected width and determine if the tolerance interval will realistically fit within the engineering specification limits. If the specification limits are narrow, n=15 will yield a tolerance interval that exceeds the limits, resulting in a validation failure.

Presenting a “Reverse-Engineered Assurance Analysis” in the technical documentation file demonstrates to the Notified Body that the manufacturer understands the statistical limitations of their study design. It shifts the regulatory conversation from “Why did you choose this sample size?” to “Given this sample size, here is the maximum statistical risk we are assuming,” which is then formally documented in the Risk Management File (ISO 14971:2019).

Usability Validation and the Probability of Use Error

A specific area where sample size justification routinely fails regulatory scrutiny is in summative usability validation (IEC 62366-1:2015). The goal of usability validation is to demonstrate that the device can be used safely and effectively by the intended users without serious use errors.

The FDA guidance on human factors engineering and the EU MDR requirements for usability engineering dictate that usability testing must sample a representative population of users. Statistical justification of this sample size estimate is uniquely challenging because use errors are ideally rare events.

If a critical use error has an acceptable upper bound of 1 in 10,000 occurrences, demonstrating this via a standard c=0 attribute sampling plan would require an astronomical sample size:

Testing 30,000 participants in a usability study is generally impractical. Therefore, statistical methods for usability testing should not rely on frequentist attribute sampling alone. Under FDA human-factors guidance and IEC 62366-1, summative usability validation is treated as a primarily qualitative exercise. This entails surfacing use errors and analysing their root causes with roughly 15 participants per user group serving as a problem-discovery heuristic, rather than a statistically powered estimate of an error rate. To strengthen the regulatory submission a biostatistician may supplement this qualitative validation with a Bayesian hierarchical model that combines the summative test data with prior data from formative studies, heuristic evaluations, and predicate device post-market surveillance. By establishing a prior distribution for the use error rate, the posterior distribution can be evaluated to provide additional quantitative assurance that the upper 95% credible bound for the critical use error rate falls below the acceptable threshold.

This approach aligns with the MDR’s emphasis on continuous benefit-risk evaluation and the integration of post-market data into the clinical evaluation cycle.

Documenting the Defence of a Sample Size Estimation

A robust sample size estimation section in a V&V protocol or CER must contain the following elements to survive Notified Body scrutiny:

1. Objective Statement: Explicitly state whether the objective is to estimate a parameter (requiring a CI), bound a future observation (requiring a prediction interval), or bound a proportion of the population (requiring a tolerance interval).
2. Statistical Framework: Specify the exact formula or distribution used. If using a c=0 plan, provide the binomial derivation. If using a Cpk plan, specify the non-central t-distribution parameters.
3. Assumptions: Detail the assumed variance, the assumed defective rate, or the prior distribution. Provide the rationale for these assumptions based on pilot data or engineering analysis.
4. Risk Integration: Map the statistical parameters (e.g., \alpha, \beta, p_0) directly to the risk acceptance criteria in the Risk Management File. If a hazard is classified as “Catastrophic,” the statistical assurance must be correspondingly high (e.g., 99% confidence).
5. Sensitivity Analysis: If Bayesian assurance is used, demonstrate the impact of varying the prior distribution. If frequentist power is used, demonstrate the impact of varying the assumed effect size.

Summary

Sample size estimation in a medical device context is a multi-faceted statistical consideration that encompasses engineering constraints, clinical risk, and regulatory expectations. By use of acceptance sampling, the derivation of non-central t-distributions for Cpk, and the Bayesian integration of prior data via assurance calculations, a sample size justification can become a quantitative asset, rather than a regulatory vulnerability.

References:

1. Pardo, S. A. (2023). Statistical Methods and Analyses for Medical Devices. Springer.
2. European Parliament and Council. (2017). Regulation (EU) 2017/745 on medical devices (MDR), Articles 61, 62, 83, and Annex XV. Official Journal of the European Union.
3. International Organization for Standardization. (1999). ISO 2859-1:1999 – Sampling procedures for inspection by attributes.
4. Squeglia, N. L. (2023). Zero Acceptance Number Sampling Plans (6th ed.). Quality Press.
5. International Organization for Standardization. (2014). ISO 16269-6:2014 – Statistical interpretation of data – Part 6: Determination of statistical tolerance intervals.
6. International Electrotechnical Commission. (2015). IEC 62366-1:2015 – Medical devices – Application of usability engineering to medical devices.
7. US Food and Drug Administration. (2016). Applying Human Factors and Usability Engineering to Medical Devices.
8. US Food and Drug Administration. (2010). Guidance for the Use of Bayesian Statistics in Medical Device Clinical Trials.