Skip to main content

statswork

Stronger Analysis. Smarter Research. Bigger Savings!
Access expert statistical analysis and research support at Flat 24% Off
Stronger Analysis. Smarter Research. Bigger Savings!
Access expert statistical analysis and research support at Flat 24% Off.

Is Your Batch-Release ML Model ‘Static’ Enough for Draft Annex 22? Validating Deterministic Models in Irish Pharma Plants

Summary

A static, deterministic machine-learning (ML) model can produce identical outputs and still approve a non-conforming batch. Irish pharmaceutical manufacturers need evidence of configuration control, execution repeatability and predictive performance. The July 2025 consultation draft proposes AI-specific GMP guidance, while the EMA’s October 2026 workshop report records stakeholder input intended to inform the ongoing revision of Annex 22. Neither should be mistaken for a final, binding Annex 22 requirement.

AI in pharmaceutical manufacturing requires risk-based validation when models support GMP-critical decisions. EU GMP Annex 22 AI validation must consider intended use, model performance and quality risk. Effective machine learning validation in pharmaceutical manufacturing must establish whether the approved model performs reliably under intended operating conditions and whether the complete application supports controlled, traceable decisions.

What does draft EU GMP Annex 22 mean?

The European Commission opened its consultation on revised GMP Chapter 4, Annex 11 and proposed new Annex 22 on 7 July 2025; it closed on 7 October 2025. The draft addresses model selection, intended use, validation, training and test data, performance monitoring and change control. These remain proposed provisions. [1]

The EMA’s multistakeholder workshop took place on 30 June and 1 July 2026, with its report published on 2 October 2026. Discussions covered adaptive and probabilistic models, guardrails, human oversight, validation and lifecycle management. The report informs guidance development but does not establish binding requirements. [2]

The July 2025 draft excludes generative AI/LLMs, probabilistic, dynamic and adaptive models from critical GMP use. The 2026 EMA workshop explored a risk-based approach to these technologies, but no final permission was established. Static, deterministic models may fit the draft’s scope more closely, but suitability and regulatory acceptance are not guaranteed.

Separate configuration, repeatability and accuracy

These are three distinct validation questions.

Property

Evidence required

What it does not prove

Static configuration

Approved version, parameters and change history

Correct predictions

Deterministic execution

Repeatable outputs under defined conditions

Performance on unseen cases

Predictive performance

Independent testing against reference outcomes

Permanent future performance

 

A model may repeatedly classify a defective batch as acceptable. Its output is deterministic, but its prediction is wrong.

A frozen model can also run within a changing application. Preprocessing, software dependencies, input systems and decision thresholds may alter results. Statistical evidence supports, but does not replace, wider GMP validation, including software lifecycle controls, data integrity, integration testing, audit trails and authorised release procedures.

Define the model’s decision authority explicitly. A model used to prioritise quality investigations presents a different risk from one whose output directly determines whether a batch may be accepted. Document how the output interacts with approved specifications, deviation handling, human review and the authorised batch-release process.

NIST’s voluntary AI Risk Management Framework supports principles of validity, reliability and representative testing, but is not a pharmaceutical GMP standard. [3]

Establish configuration control

Document the model artefact, version, learned parameters, preprocessing logic, thresholds and relevant software dependencies. Cryptographic hashes can help detect model-file changes but do not prove that the complete deployed application remains unchanged or scientifically valid.

Restrict update permissions, retain approved versions and document change authorisation. Retraining, parameter updates and preprocessing changes should trigger impact assessment and applicable change control and revalidation.

Maintain traceability from intended use and risk assessment through requirements, test cases, acceptance criteria, deviations and final approval.

Test repeatability and independent performance

Validate the model through a structured workflow covering configuration control, repeatability, independent performance testing, statistical risk assessment and ongoing monitoring, as illustrated below.

Test the complete application pathway, not just model inference. Repeatedly running one input 2,000 times cannot demonstrate performance across 2,000 independent production cases.

For GMP batch-release model validation, freeze the model and analysis protocol before examining final test results. Keep test data independent of training and model selection, and assess leakage involving related batches, products, sites and time periods.

Choose metrics according to the consequences of incorrect decisions. Sensitivity measures the proportion of genuinely non-conforming cases detected; specificity measures the proportion of conforming cases correctly identified. Report confidence intervals and prespecified acceptance criteria. FDA, Health Canada and MHRA Good Machine Learning Practice principles provide supplementary methodological guidance but concern medical devices rather than pharmaceutical manufacturing. [4]

Quantify false-negative risk

Suppose an independent test set contains 200 non-conforming cases, of which the model flags 190.

“Sensitivity”=190/200×100%=95%

The observed false-negative rate is 10/200 = 5%. If an unflagged case can proceed to batch acceptance, these misses represent potential false acceptances.

The approximate 95% Wilson confidence interval for sensitivity is 91.0%–97.3% [5]. If a predefined criterion requires a lower confidence bound of at least 93%, the model fails despite its 95% point estimate.

This is an illustrative rule, not an Annex 22 threshold. Acceptance criteria must be justified by intended use and quality risk and specified before final testing.

Sensitivity alone does not establish the probability that an accepted batch is non-conforming. That probability also depends on prevalence, specificity and the decision pathway. Analyses must account for dependence when observations are clustered by batch or production line.

Assess subgroups and failure conditions

Pooled performance can conceal weaknesses. Where relevant, evaluate performance by product or strength, production line, equipment, raw-material characteristics, operating conditions and defect type.

Validation finding

Appropriate response

Configuration mismatch

Block use pending investigation

Repeatability failure

Investigate execution

Performance criterion missed

Investigate errors and revalidate as justified

Subgroup weakness

Restrict scope or validate mitigation

Monitoring alert breached

Assess impact and escalate

 

Justify subgroup selection using intended use, process variation and quality risk. Where sample sizes are insufficient, report uncertainty rather than assuming acceptable performance.

Define responses to missing values, out-of-range inputs and low-confidence predictions. Depending on the validated design, these may require input rejection, escalation or authorised human review. Do not relax acceptance criteria retrospectively to obtain a pass.

Maintain control after deployment

Monitor input distributions, missing-data rates, output patterns and performance when reference outcomes become available. Distinguish input drift from demonstrated deterioration in predictive performance.

Define monitoring frequency, alert thresholds, escalation responsibilities and suspension criteria. Link these controls to pharmaceutical quality risk management, applicable computerised-system requirements and data-integrity controls, including relevant Annex 11 provisions.

Assess your batch-release ML validation readiness

Statswork’s Data Analysis service supports Irish pharmaceutical manufacturers with statistical evidence for risk-based machine-learning validation. Support can include:

  • Validation study design: Prespecified metrics, acceptance criteria and analysis plans.
  • Independent test-data assessment: Sample-size adequacy, representativeness, dependence and potential information leakage.
  • Statistical performance evaluation: Repeatability, sensitivity, specificity, confidence intervals and false-negative risk.
  • Subgroup risk analysis: Performance weaknesses across manufacturing conditions.
  • Monitoring strategy: Statistical justification of alert thresholds and escalation criteria.

A structured readiness assessment can review intended use, available test data, acceptance criteria, subgroup coverage and monitoring plans. Agreed outputs may include a gap summary, prioritised findings, further-testing recommendations and a validation protocol with reproducible calculations and performance tables.

Assess your batch-release ML validation readiness

Reach Statswork’s data analysis service to discuss your model’s intended use, test-data coverage and statistical validation requirements. The assessment can inform validation and quality-review activities but does not guarantee regulatory acceptance or replace the manufacturer’s responsibility for GMP compliance and authorised batch-release decisions.

Conclusion

Validating a static ML model requires more than repeatable outputs. Manufacturers need evidence of configuration control, consistent execution, acceptable predictive performance on independent data and ongoing lifecycle oversight. For Irish pharmaceutical plants preparing for evolving Annex 22 guidance, risk-based statistical validation makes model behaviour measurable, traceable and assessable for its intended GMP use.

Frequently Asked Questions

The July 2025 consultation draft proposes controls and restrictions for AI in critical GMP applications. The final applicable regulatory position must be established from official guidance.

No. Repeated execution tests reproducibility; predictive performance requires evaluation against appropriate reference outcomes.

Sensitivity, specificity, false-negative rate and confidence intervals may be relevant. Criteria should reflect the consequences of incorrect decisions.

Yes. Parameters can remain unchanged while incoming data or manufacturing conditions change.

Statswork can support validation study design, independent test-data assessment, performance analysis, subgroup evaluation and monitoring criteria through its Data Analysis service.

References:

  1. European Commission. (2025). EudraLex Volume 4: Good manufacturing practice guidelines—Annex 22: Artificial intelligence (draft). https://health.ec.europa.eu/document/download/5f38a92d-bb8e-4264-8898-ea076e926db6_en?filename=mp_vol4_chap4_annex22_consultation_guideline_en.pdf
  2. European Medicines Agency. (2026, October 2). Report: Multistakeholder workshop on expert contributions to artificial intelligence guidance development (Annex 22). https://www.ema.europa.eu/en/documents/report/report-multistakeholder-workshop-expert-contributions-artificial-intelligence-guidance-development-annex-22_en.pdf
  3. Tabassi, E. (2023). Artificial intelligence risk management framework (AI RMF 1.0) (NIST AI 100-1). National Institute of Standards and Technology. https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf
  4. U.S. Food and Drug Administration, Health Canada, & Medicines and Healthcare products Regulatory Agency. (2021, October). Good machine learning practice for medical device development: Guiding principles. https://www.fda.gov/medical-devices/artificial-intelligence-enabled-medical-devices/good-machine-learning-practice-medical-device-development-guiding-principles
  5. Newcombe, R. G. (1998). Two-sided confidence intervals for the single proportion: Comparison of seven methods. Statistics in Medicine, 17(8), 857–872. https://doi.org/10.1002/(sici)1097-0258(19980430)17:8%3C857::aid-sim777%3E3.0.co;2-e

Contact us