Human Oversight¶

Level: Intermediate · Time: ~35 min · Exam relevance: CCAR-P D5 (5.3, 5.1), CCAO-F D2 (2.4), CCAO-F D6 (6.3)

In plain English¶

Human oversight means a person can understand, check, stop or overrule what an AI system does, at the points where it matters. It is not "a human clicks OK on everything". Oversight that is badly designed creates a false sense of safety: reviewers approve by reflex, and the organisation believes a control exists when it does not.

Good oversight is proportionate (more for higher-risk actions), informed (the reviewer sees what they need to judge), and measurable (you can tell whether reviewers are actually catching problems).

Module 10: human in the loop shows how to build approval steps into an agent loop. This page is about designing oversight as a security and governance control.

Why it matters¶

CCAR-P-5.3 is "apply human-in-the-loop validation"; CCAO-F-2.4 asks when human review or extra verification is needed.
Human oversight is an explicit requirement for high-risk systems under the EU AI Act and appears throughout the NIST AI RMF (see AI governance). NIST AI 600-1 lists human-AI configuration, including over-reliance and automation bias, as a generative AI risk.
It is the compensating control for the residual risk you cannot remove technically, for example a convincing but wrong recommendation.
The agentic risk ASI09 Human-Agent Trust Exploitation describes agents (or attackers steering them) exploiting the trust humans place in them, which badly designed approvals invite.

How it works¶

Oversight patterns¶

Pattern	Description	Best for
Human-in-the-loop (HITL)	A human approves before the action happens	Irreversible, external or high-value actions
Human-on-the-loop (HOTL)	The system acts; a human monitors and can intervene or roll back	High-volume, reversible actions
Human-in-command	Humans set goals, limits and can stop the system at any time	All agentic systems (kill switch, scope limits)
Sampled review	Humans review a statistical sample of outputs after the fact	Quality assurance, drift detection
Escalation	The system routes uncertain or sensitive cases to a human	Classification, triage, customer support

Deciding where humans go¶

Use the risk tiers from tool misuse and excessive agency and add three questions per action:

Reversibility: can it be undone cheaply and completely?
Blast radius: how many people, records or how much money does it affect?
Verifiability: can a human actually judge correctness in the time available?

If an action is irreversible and high blast radius, require HITL. If it is reversible, prefer HOTL with good monitoring. If a human cannot verify it in the time available (for example a 2,000-line code change reviewed in 30 seconds), the approval is theatre: restructure the task so that review is feasible (smaller changes, automated tests, diffs, evidence).

Designing the approval moment¶

What the reviewer sees decides whether the control works.

Show the exact action and parameters (recipient, amount, record ID, diff), generated by code from the tool call, not the model's description of it. A manipulated model can describe a harmful action innocently.
Show provenance: which sources and tool results led here, and flag any untrusted content involved.
Show what changes: before and after values, or a diff.
Make reject and edit as easy as approve, and require a reason for overrides of high-risk flags.
Avoid batch "approve all" for tier-3 actions.
Time-box approvals and expire stale ones.

Countering automation bias¶

People tend to over-trust confident automated suggestions. Mitigations:

Display uncertainty and the evidence, not just the answer.
Occasionally insert known-bad "canary" cases (clearly tracked) and measure whether reviewers catch them.
Track approval rate, time-to-decision and override rate per reviewer; a 100 percent approval rate with a four-second median decision time is a red flag.
Rotate reviewers and keep queues manageable.
Train reviewers on the system's known failure modes.

Example¶

The approval request is built by code from the tool call, so the model cannot shape what the reviewer sees.

"""approval.py - build reviewer-facing approval requests from tool calls, not model prose."""
import json
import time
import uuid
from dataclasses import dataclass, field

@dataclass
class ApprovalRequest:
    request_id: str
    tool: str
    exact_args: dict
    before: dict
    after: dict
    untrusted_sources: list[str]
    risk_flags: list[str]
    expires_at: float
    model_rationale: str = field(default="")  # shown last and labelled as model-generated

def build_request(tool_use: dict, current_record: dict, sources: list[dict]) -> ApprovalRequest:
    args = tool_use["input"]
    after = {**current_record, **{k: v for k, v in args.items() if k in current_record}}
    flags = []
    if any(s["trust"] == "untrusted" for s in sources):
        flags.append("Untrusted content was in context when this action was proposed")
    if args.get("amount_cents", 0) > 50_000:
        flags.append("Amount above 500.00")
    return ApprovalRequest(
        request_id=str(uuid.uuid4()),
        tool=tool_use["name"],
        exact_args=args,
        before=current_record,
        after=after,
        untrusted_sources=[s["id"] for s in sources if s["trust"] == "untrusted"],
        risk_flags=flags,
        expires_at=time.time() + 15 * 60,
    )

def record_decision(req: ApprovalRequest, reviewer: str, decision: str, reason: str, audit) -> None:
    if time.time() > req.expires_at:
        raise TimeoutError("approval expired; re-run the proposal")
    if decision == "approve" and req.risk_flags and not reason.strip():
        raise ValueError("a reason is required to approve a flagged action")
    audit.write("approval_decision", request_id=req.request_id, reviewer=reviewer,
                decision=decision, reason=reason, tool=req.tool, args=req.exact_args,
                flags=req.risk_flags)

if __name__ == "__main__":
    req = build_request(
        {"name": "update_credit_limit", "input": {"account_id": "AC-1029", "credit_limit_cents": 120_000}},
        current_record={"account_id": "AC-1029", "credit_limit_cents": 80_000},
        sources=[{"id": "email:5512", "trust": "untrusted"}, {"id": "policy:limits-v4", "trust": "internal"}],
    )
    print(json.dumps(req.__dict__, indent=2))

audit is the logger from logging and auditability. Lab 13: human in the loop builds a full approval flow.

Enterprise use case¶

Contoso Health uses an agent to draft prior-authorisation responses. Initially every draft needed a nurse's approval. After a month, data showed a 99.6 percent approval rate and a median review time of six seconds, with two incorrect approvals found in a sampled audit.

The redesign:

Routine, low-risk categories with strong eval scores move to HOTL: sent automatically, sampled at 5 percent, with a 48-hour recall window.
Denials and any case with untrusted attachments stay HITL, and the review screen shows the cited policy clauses next to the relevant record fields.
Two tracked canary cases per reviewer per week; catch rate is reported monthly.
The median review time for HITL cases rises to 70 seconds and incorrect approvals in sampling drop to zero over the next quarter.

Common mistakes¶

Approval screens that show the model's summary instead of the exact action.
Requiring approval for everything, which trains reviewers to click through.
No measurement of reviewer effectiveness.
No way to stop a running agent or roll back its actions.
Treating "a human was in the loop" as a substitute for technical controls.

Security considerations¶

Approval channels are an attack surface: authenticate reviewers, protect against approval forgery (the model must never be able to set an "approved" flag) and log decisions with reviewer identity.
Attackers may try to induce approvals with urgency or authority language in content the model reads. Flag untrusted sources on the approval screen.
Separation of duties: the person who requested an action should not be the only approver for high-risk actions.

Troubleshooting¶

Symptom	Likely cause	Fix
Approval queue grows faster than reviewers can clear it	Too many actions in HITL	Move reversible actions to HOTL; improve evals to justify automation
Reviewers miss canary cases	Fatigue, unclear UI, automation bias	Smaller queues, clearer diffs, training, rotation
Users complain the agent is slow	Synchronous approvals in user-facing path	Asynchronous approval with notifications

Check your understanding¶

What is the difference between human-in-the-loop and human-on-the-loop?
Why must the approval screen be generated from the tool call rather than from the model's explanation?
Name two metrics that reveal ineffective human review.
When is a human approval step "theatre"?

Answers

HITL requires human approval before an action; HOTL lets the system act while a human monitors and can intervene or roll back.
Because a mistaken or manipulated model could describe a harmful action as harmless; code-generated views show what will actually happen.
Any two of: very high approval rate, very short median decision time, low canary catch rate, low override rate despite known error rates.
When the reviewer cannot realistically verify the action in the time and with the information available, so approval adds no real check.