Implementing AI/ML in GMP: Guardrails as Risk Controls
One of the central topics discussed during the EMA Multistakeholder Workshop on Annex 22 was the use of guardrails for AI/ML systems, particularly probabilistic models such as Large Language Models (LLMs). The importance of this concept is reflected in the workshop agenda itself: across six discussion topics and 20 questions raised by the regulators, the EMA explicitly referred to guardrails eleven times.
What are Guardrails?
One of the first challenges during the preparation of the industry response was surprisingly simple: What exactly is a guardrail? While the term is widely used, there is currently no universally accepted definition, and different stakeholders interpret it differently.
In the GMP context, guardrails can be understood as technical control mechanisms that reduce the risks associated with AI/ML systems. Rather than relying solely on procedural controls, guardrails improve the reliability and trustworthiness of model outputs through technical measures.
During an excellent industry presentation by Toni Manzano (CEO, aizon) on behalf of PDA, guardrails were grouped according to the stage at which they operate:
Input Guardrails – Prevent invalid or malicious inputs.
In-Model Guardrails – Detect issues during model execution.
Output Guardrails – Contain or mitigate undesirable outputs before they influence GMP decisions.
Guardrails can be effective risk control measures with the potential to reduce risks to an acceptable level. Their application and life cycle controls should be governed under Quality Risk Management (QRM).
The Swiss Cheese Model
The concept aligns well with the Swiss Cheese Model, where multiple independent control layers reduce overall risk. Failure occurs only when weaknesses in every control layer align, allowing a hazard to pass through all defenses.
The key principle is independence. Just as slices from different cheeses would have different hole patterns, independent guardrails are more likely to detect different failure modes than multiple controls based on the same mechanism. This concept has already been proposed for AI/ML controls in GxP by Victor Bechmann et al. [1].
Practical Examples
Examples presented during the workshop included:
Examples of different guardrail types presented at the workshop.
Conclusion
The workshop discussion reinforced that guardrails should be viewed through the principles of ICH Q9(R1): they are risk control measures that reduce the likelihood or impact of AI/ML failures.
Because different guardrails address different risks, there is no one-size-fits-all solution. Instead, the guardrail architecture should be selected based on the identified risks and continuously reviewed throughout the AI/ML system lifecycle. As knowledge of the process and model evolves, guardrails should be adapted or extended to maintain an acceptable level of risk.
Stay tuned and sign up for more content here.
