Human-AI Interaction & Oversight: NCP-AAI Domain 10 (5%)

What this domain covers

Oversight is a design decision, not a disclaimer. This domain asks where a human belongs in the loop, what they need in order to make a real judgement, and how the system behaves while it waits for them. It is the counterpart to the safety domain: safety constrains what an agent may do, oversight decides who confirms it.

Placing the gate: risk, not uniformity

Requiring approval for every step destroys the value of automation, and requiring it for none removes the control. The usual answer is to gate on consequence: let reversible, low-impact actions run autonomously, and require confirmation for actions that are irreversible, costly, externally visible or that touch sensitive data. The same agent can be fully autonomous on reads and gated on writes.

Autonomous   read, search, retrieve, draft, summarise
Gated        send, pay, delete, publish, modify a record
Escalate     low confidence · repeated failure · out of policy scope

Making review meaningful

An approval prompt that shows only a proposed action produces rubber-stamping — the reviewer has no basis to disagree. A useful prompt shows what the agent intends to do, why, what evidence it relied on, and what happens if it is wrong. Surfacing the reasoning trace and its sources is what converts an approval click into an actual decision.

Escalation and graceful handover

Agents need a defined route for cases they should not finish: low confidence, repeated tool failure, a request outside policy, or explicit user demand for a human. The handover matters as much as the trigger — the human should receive the full context rather than a cold start, and the user should not have to repeat themselves. Escalation rate is also a quality signal worth monitoring: a sharp rise usually means something upstream has degraded.

Transparency toward the user

People interacting with an agent should know they are, should be able to tell what it has done on their behalf, and should have a way to interrupt or correct it. Uncertainty is worth surfacing rather than hiding — an agent that says it is unsure and offers a human is more trustworthy, and usually more useful, than one that answers confidently and wrongly.

Exam tip

When a question asks where to put a human in the loop, sort the options by reversibility. The correct design almost always gates irreversible or externally visible actions and leaves reversible ones autonomous — not "approve everything" and not "approve nothing".