Deployment & Scaling: NCP-AAI Domain 4 (13%)

What this domain covers

Deployment and Scaling is about running agents in production: how you serve them, keep latency and cost under control, handle concurrent users, and scale as load grows. Agentic workloads are unusually demanding because a single user request can fan out into many model and tool calls. This domain is 13% of the exam.

Why agents are expensive to serve

A single agent task is not one model call — it is a loop of many calls, each carrying a growing context of prior steps and tool results. Cost and latency compound with every step. The core deployment skill is controlling that fan-out: capping steps, trimming context, caching, and choosing the right-sized model per step.

# One user request can become many model + tool calls
request → [plan → tool → observe] × N steps → answer
# Cost ≈ Σ (input_tokens + output_tokens) over every step
# Latency ≈ Σ (model_latency + tool_latency) over every step

Latency and cost controls

Practical levers include: streaming responses so users see progress; running independent tool calls in parallel rather than sequentially; caching retrieval and repeated calls; using a smaller/faster model for easy sub-steps and a larger one only where needed; and setting hard step/token budgets per request.

Stream output ........ perceived latency ↓
Parallel tool calls .. wall-clock latency ↓
Cache retrieval ...... repeated cost ↓
Right-size the model . per-step cost ↓
Step/token budgets ... worst-case cost capped

Statelessness and concurrency

To scale horizontally, the serving layer should be stateless — conversation and agent state live in an external store (cache or database) keyed by session, not in process memory. That lets any instance handle any request and lets you add instances behind a load balancer as traffic grows.

Reliability under load

Production agents must degrade gracefully: apply rate limiting and backpressure, set timeouts on every model and tool call, retry transient failures with backoff, and have a fallback path when a dependency is down. Autoscaling handles volume, but only if the system is stateless and each call is bounded.

Exam tip

Scaling questions usually reduce to two ideas: keep the serving layer stateless (state in an external store) and bound the fan-out (cap steps/tokens, parallelize and cache). If an option keeps agent state in process memory or lets the loop run unbounded, it is almost always the wrong answer.