Hello everyone! Welcome back to part 4 of our multi-part blog series exploring ISO 42001:2023, the Artificial Intelligence Management System (AIMS) standard. Today, we’re diving into Clause 9: Performance Evaluation and Clause 10: Improvements. If you haven’t had a chance yet, catch up on the earlier parts here: Part 1 — Introduction and Clause 4, Part 2 — Clause 5 & 6, and Part 3 — Clause 7 & 8.
Clauses 9 and 10 are the last clauses in the standard and represent the check and act of the PDCA lifecycle. Let’s dive right in.
Clause 9 — Performance Evaluation
Clause 9.1 — Monitoring, measurement, analysis, and evaluation
To understand this clause, let’s start with the definitions. ISO’s official definitions are as follows:
Monitoring — to determine the status of a system, a process, or an activity.
Measurement — a process to determine value
Now translating this into ISO 42001 requirements, this clause requires you to determine what you need to monitor and measure, how you will do that (like the methods you will use), when you will do this activity, and how you will use the results (read: analyze and evaluate).
It’s important to understand that monitoring and measurement extend far beyond just controls, which is a common misconception. The scope encompasses monitoring of your organizational objectives, the effectiveness of your risk management system, and potentially any clause within the AIMS framework. The focus areas ultimately depend on what’s most important for your specific organization and AI application context.
If implemented incorrectly, several negative consequences can arise, such as:
- Ineffective Risk Management: Without proper monitoring metrics, you might miss critical risks in your AI system’s behavior, potentially leading to harmful outputs or decisions.
- Compliance Violations: Inadequate performance tracking could lead to violations of regulatory requirements resulting in penalties and legal exposure.
- Improvement Stagnation: Without meaningful metrics and analysis, you cannot identify areas for improvement, leading to an AI system that fails to evolve with changing needs and conditions.
These consequences collectively undermine the entire purpose of the AI Management System (AIMS), ruin the organization’s ability to effectively govern its AI, and potentially damage the organizational reputation.
To address these challenges, here are several practical examples for our AI-powered medical diagnosis support system to help you develop an effective approach. To recap from Part 1, we are looking at a fictitious organization that is a private multi-specialty hospital implementing an AI-powered diagnosis support tool to assist clinicians in interpreting radiology scans. They can put some of the following monitoring and measurements in place to ensure effective AIMS.
Clause 9.2 — Internal Audit
This requirement is similar to other management system standards. Conduct internal audits at regular intervals to verify that the AI management system meets both the organization’s own defined requirements and the standard ISO 42001.
An important aspect of internal audit is that it must be impartial and objective. For this reason, the audit team should ideally be independent from those directly responsible for the audited activities. This organizational separation helps ensure unbiased assessment and meaningful findings that can drive genuine improvement in your AIMS implementation.
The clause requires maintaining documented information about the audit program, evidence of implementation of the audit, and the audit results.
Clause 9.3 — Management Review
The clause requires you to conduct management reviews at regular intervals and cover the input requirements detailed in clause 9.3.2, while maintaining evidence of results as defined in clause 9.3.3. The frequency of reviews should align with the pace of change in your AI systems and their operating environment.
While this requirement shares similarities with other management system standards, it plays a particularly crucial role and a strategic checkpoint in AI governance. Management reviews serve as the primary mechanism where senior leadership evaluates the effectiveness of the entire AIMS and makes strategic decisions about its future direction.
The key to effective management reviews lies not in treating them as compliance exercises, but as genuine opportunities to assess whether your AI systems are delivering value while managing risks appropriately.
Clause 10 — Improvements
Clause 10.1 — Continual Improvement
This clause emphasizes the ongoing effort to improve the suitability, adequacy, and effectiveness of AI Management System (AIMS). It’s not just about fixing problems; it’s about proactively seeking ways to make the management system better over time. This involves leveraging the insights gained from monitoring, measurement, analysis, and evaluation (Clause 9), internal audits (Clause 9.2), and management reviews (Clause 9.3). The key is to analyze data, feedback, audit results, and changes in context, take action to improve, and check if the changes led to the desired outcomes. It is a continuous cycle and hence called continual improvement.
Clause 10.2: Nonconformity and Corrective Action
This clause outlines the process for addressing situations where the AIMS or its outputs do not meet specified requirements. Nonconformity can be identified during an audit, a management review, an external audit, or by anyone in the organization. It’s about reacting to a problem by identifying, analyzing, and correcting nonconformities to prevent recurrence.
As far as documented information goes, the clause asks you to maintain evidence of the nature of nonconformity, action taken, and the result of the corrective action. Here is a flowchart to explain this clause.
This was our final clause for this standard and the end of the standard requirements. Hope this gives you a sense of the management aspect of the standard. The final installment of our multi-part blog will focus on Annex A Controls of the standard, which provides organizations with a reference for meeting the objectives and addressing the risks related to the design and use of AI systems.
References
- **ISO 42001 — **Information technology — Artificial intelligence — Management system: https://www.iso.org/standard/81230.html
- **ISO/IEC 22989 — **Information technology — Artificial intelligence — Artificial intelligence concepts and terminology: https://www.iso.org/standard/74296.html
Disclaimer
The content provided in this blog series is for informational purposes only and does not constitute legal, regulatory, or professional advice. While every effort has been made to ensure accuracy, readers are encouraged to consult the official ISO 42001 standard and relevant regulatory or industry experts for specific guidance. The views expressed are those of the author and do not necessarily reflect the opinions of any affiliated organizations.