Run, Monitor & Maintain: NCP-AAI Domain 8 (5%)

What this domain covers

Once an agent is serving traffic, the questions change from "does it work?" to "is it still working, how do I know, and what do I do when it stops?" This domain covers observability built for multi-step systems, the cost and latency telemetry agents uniquely need, detecting quality regressions that no exception ever reports, and the maintenance loop that follows.

Tracing is the primitive

A single agent request is not one call — it is a tree of model calls, tool invocations and retrievals, any of which can be the thing that went wrong. Logging only the final response makes failures undebuggable. The unit of observability is a trace: one span per step, carrying inputs, outputs, latency, tokens and errors, all tied to a request id so a complaint can be traced to the exact step that failed.

trace  request_id=4f21c
 ├─ span  plan            420ms   1,180 tok
 ├─ span  tool:search     310ms   — (3 results)
 ├─ span  retrieve        180ms   — (5 chunks)
 ├─ span  synthesise      890ms   2,340 tok
 └─ span  guardrail:out    40ms   pass

The metrics agents need that ordinary services do not

Standard service monitoring — error rate, p95 latency, throughput — still applies, but misses what actually goes wrong with agents. Track tokens and cost per request, because an agent that starts looping is a cost incident before it is an outage. Track steps per task, because rising step counts signal degrading planning. Track tool failure and retry rates, and track how often guardrails fire.

Standard:  error rate · p95 latency · throughput
Agentic:   tokens/request · cost/request · steps/task
           tool error rate · retry rate · guardrail trigger rate
           escalation-to-human rate

Quality drift, and why it is silent

An agent can degrade while every system metric stays green: a model version changes, a knowledge base goes stale, a tool alters its response shape, or real user inputs drift away from what you tested. None of these raise exceptions. Catching them needs quality signals rather than health signals — a fixed evaluation set run continuously against production, sampled human review, and user feedback captured in-product.

Maintenance is a loop, not an event

The operating cycle is: observe production traces, find the failure patterns, turn each into a regression case in the evaluation set, fix, and re-run. Over time the evaluation set becomes an accumulated record of every way the system has actually failed — which is far more valuable than a set written from imagination before launch. Roll changes out behind canaries so a regression reaches a fraction of traffic, and keep rollback cheap.

Exam tip

When a question describes an agent that is "working" but producing worse answers over time with no errors in the logs, it is testing quality drift. The answer is a continuous evaluation signal — not more infrastructure monitoring, which would have stayed green throughout.