This article is published in collaboration with Binaire, the blog to understand the challenges of digital.
An AI system must always be supervised by a human, but it is crucial that this person can distinguish when they understand what the machine is proposing and when they can be influenced.
Contemporary governance frameworks for artificial intelligence (AI) are based on a rarely explicit assumption: when a human operator receives the output of an AI system, they must be able to evaluate it meaningfully. The provisions of the European AI Act on high-risk systems require transparency, explainability, and human supervision.
Explicitly targeted systems include those used in recruitment and worker evaluation, access to social benefits, credit decisions, border control, justice administration, and critical healthcare.
The US AI action plan calls for meaningful human control over AI decisions with significant consequences. The OECD’s AI principles place human-centric values at the core of its commitments.
These commitments are necessary but insufficient. They focus on what AI systems should provide to human operators, leaving unanswered the question of what these individuals must be capable of doing to act on what they receive. This gap is not accidental. It is a structural blind spot in the current architecture of AI governance.
The implicit model of a human supervisor in most regulatory texts is that of a competent and attentive professional who, faced with precise and readable outputs, formulates clear judgments. This is a plausible assumption in stable, low-stakes, and well-understood environments, but a fragile one in high-stakes contexts, under time pressure, and technically opaque – precisely the contexts in which AI systems are increasingly deployed.
For example, the emergency room nurse in charge of triage who receives a triage score generated by an AI system may not always have the explanations they need. The bank advisor who must decide within minutes to block an account based on an automated fraud alert potentially works with a model that cannot be questioned. The administrative officer who approves social housing allocation or algorithmically prioritized benefits generally cannot explain why one application was processed before another. The teacher who countersigns an automated exam grading does not have access to the criteria that produced the score. In each of these cases, human supervision is formally present but substantively impossible.
Operators Metacognitively Aware
Metacognition – the ability to monitor and regulate one’s own cognitive processes – is the psychological substrate of effective supervision. A metacognitively aware operator knows when they understand something, when they speculate, and when their judgment is influenced by factors they are not consciously aware of. This capacity cannot be presumed; it varies significantly among individuals, training, and situational pressures.
Research on human-automation interaction has documented a set of failure modes that specifically emerge when humans supervise AI-powered or automated systems. Automation bias – the tendency to overweight machine recommendations compared to one’s judgment – is one of the most robust findings in the field. In a frequently cited study, researchers Parasuraman and Riley showed in 1997 that humans systematically misuse automation – applying it where it is less reliable and disregarding it where it would be beneficial – reflecting a lack of metacognitive calibration rather than information provision. For example, in flight simulator experiments cited by these authors, pilots equipped with an automated alert system shut down an engine in response to a false alert – a decision they had agreed, before the experience, never to make based solely on an automated alert.
The challenge is compounded by the features of contemporary AI systems. Kahneman’s work on dual-process cognition – also known as System 1/System 2, the two speeds of thinking – illuminates this mechanism. Faced with an AI system that produces an output fluently and confidently, the human mind tends to engage in rapid and intuitive processing (the one mobilized for familiar and low-risk tasks) rather than engaging in a deeper, slower, more thoughtful, more logical analysis of the situation, which is more cognitively demanding.
Three Implications for AI Governance
If metacognitive maturity is a real and variable property of human operators, then governance frameworks that impose explainability without considering operators’ metacognition are simply incomplete. According to scientific literature – including explainable AI, human-automation interaction, cognitive science, psychology, and social sciences – three implications can be articulated:
– Transparency focusing on documentation is insufficient. It’s not just an intuition: research has shown this for thirty years. Thus, documenting and explaining the behavior of a system is not enough to ensure good human decisions without involving individuals in the design processes of these explanations and documentation and considering the context of the business need at that time. Controlled studies have even shown that “too many explanations” can degrade the performance of the human-AI team by drowning out relevant information in noise.
– The metacognitive qualification of operators should be considered as a component of AI governance. This is a gap that research has begun to identify, but no reference has yet been formalized.
In concrete terms, regulatory texts like the AI Act require that human supervisors be “competent,” but without ever defining what that means – and notably, no reference evaluates what researchers call metacognitive competence, the ability to detect flaws in one’s own reasoning facing an opaque system, a competence that derives from training and context, not raw intelligence. An important clarification is needed here. Speaking of the metacognitive qualification of operators is not about questioning the value or intelligence of the people supervising AI systems. It is also not about classifying humans according to their ability to “think well.” Metacognition is neither a personality trait nor an indicator of value. It is a situational competence, sensitive to context, training, cognitive load, and working conditions. For example, an experienced surgeon may have excellent metacognitive calibration in their field but may be just as vulnerable to automation bias as a beginner facing an opaque AI system in a context for which they have received no specific training.
