AI agent interview deep dive

No — the user's raw message is actually the smallest piece of what gets sent. On every single call, the code layer assembles a full context window made up of five things:

• System prompt — the agent's standing instructions: its role, tone, rules it must follow, and boundaries (what it's not allowed to do). This is set by the developer and never shown to the user directly.
• Tool schemas — the full definitions of every tool currently available to the agent: name, description, and parameters. This is re-sent on every call because the LLM has no memory between API calls — it doesn't "remember" the tools from last time, so they're re-declared every request.
• Conversation history — the recent back-and-forth of the current session, sometimes trimmed or summarized to save tokens once it gets long.
• Retrieved long-term memory — relevant facts pulled from the vector store based on the current message (this is what "RAG," retrieval-augmented generation, actually refers to). It's not the whole memory store — just the top few most relevant chunks.
• The current user message — the actual new thing the user just said or the result of the last tool call, if this is a mid-loop turn.

All five get concatenated (in a specific structured format — usually a list of role-tagged messages) into one prompt, sent as a single API call. The LLM then reasons over the whole thing at once and produces either a text reply or a structured tool call.

It doesn't remember anything on its own — every single API call is stateless from the model's point of view. What looks like memory is actually the code layer re-sending the relevant history every single time. As a conversation grows, sending the full transcript gets expensive and eventually exceeds the model's context window, so real systems use a few techniques:

• Sliding window — only keep the last N messages verbatim, drop older ones.
• Rolling summarization — periodically ask the LLM itself to compress older turns into a short summary, and keep that summary instead of the full text.
• Retrieval instead of replay — for very long-running relationships (weeks/months), don't try to keep the whole history in context at all; store it in the vector store and retrieve only what's relevant to the current message.

Because the LLM can only reason over what's actually in front of it — if the assembled context is missing the right memory chunk, includes an irrelevant one, or the tool schema is vague, the model's decision quality drops, even though the model itself hasn't changed at all. This is why teams spend a lot of effort on "context engineering" — deciding exactly what to include, in what order, and how much of it — as a separate discipline from prompt writing.

Every tool is registered with a name, a plain-English description of what it does and when to use it, and a strict parameter schema (usually JSON Schema). All of these schemas are sent to the LLM as part of the context window on every call — the model doesn't have them memorized, they're re-declared every time. The LLM then does semantic matching: it reads the user's intent, compares it against every available tool's description, and picks the closest match. It outputs this decision not as free text but as a structured object — a tool name plus a set of arguments — because the model has been specifically trained (fine-tuned) to produce that format when it decides an action is needed instead of a direct answer.

The accuracy of this step depends almost entirely on how well the tools are described. Two vague, overlapping descriptions will genuinely confuse the model — this isn't a hypothetical, it's one of the most common real-world agent bugs.

MCP (Model Context Protocol) standardizes how a tool exposes itself to any LLM-based agent — instead of every company writing a bespoke integration for every pair of (agent framework, tool), a tool built as an MCP server can be plugged into any MCP-compatible agent with no custom glue code. It defines a consistent way to list available tools, describe their schemas, and invoke them, over a standard transport.

The practical benefit shows up at scale: if you have 5 internal tools and 3 different agent frameworks across your company, without a standard you'd write 15 integrations; with MCP, you write 5 MCP servers and every framework can use all of them.

The code layer should never pass LLM output straight through to execution. Before running anything, it validates the returned arguments against the tool's declared schema — right types, required fields present, values within allowed ranges. If validation fails, the code layer doesn't crash the whole request; it typically returns a structured error back into the conversation ("invalid parameter: date must be in the future") so the LLM can self-correct and retry with better arguments on the next turn.

Prompt injection is malicious text — from a user, an uploaded document, or a webpage the agent reads — crafted to look like an instruction and override the agent's real behavior. It's dangerous specifically because the model can't cryptographically tell the difference between "the developer's trusted system prompt" and "text that merely looks like an instruction" — both arrive as tokens in the same context window.

A layered defense in practice looks like:

• Structural separation — clearly tag which part of the prompt is system-level, and instruct the model (and enforce via the model's own instruction hierarchy, e.g. Bedrock Guardrails or the model's system-role weighting) that content inside "user" or "tool result" roles is data, never a new instruction.
• Input scanning — a lightweight classifier or set of heuristics that flags suspicious patterns ("ignore previous instructions," "you are now...") before that content even reaches the main model.
• Least-authority tool design — even if injection succeeds in changing what the model "wants" to do, if the available tools are narrowly scoped, the actual damage is capped.
• Output-side checks — validate that the model's chosen action still matches the original user request's intent, not something introduced by fetched content.

Least privilege means each tool's underlying credentials are scoped to exactly the actions it needs — nothing broader "just in case." Concretely this means writing a dedicated IAM role per tool (or per tool category) with a policy that allows only specific actions on specific resources, rather than reusing one broad application role for every tool the agent has access to. It also means separating read tools from write/delete tools at the credential level, so a compromised or confused agent calling a "read" tool physically cannot cause a destructive side effect.

The approval request needs to show the human reviewer enough context to make a real decision quickly — what action, on what resource, with what parameters, and why the agent decided to take it — not just a bare "approve/deny" button. You'd also want to set a default-deny timeout (if nobody responds in time, the action doesn't happen automatically), and log every approval/rejection so you can later measure whether approvals are being rubber-stamped (which tells you the tool shouldn't have needed approval in the first place, or the risk framing is wrong).

Data poisoning is an attacker getting false or malicious content written into long-term memory so that it's later retrieved and treated as trusted fact in a future conversation — effectively a delayed prompt injection that persists across sessions. The defense has to happen at write time, not just read time, because by the time you're reading poisoned data back, the damage window has already opened.

• Restrict the writer — raw user text should never go straight into the vector store; only a controlled extraction step writes to it, after summarizing and filtering.
• Content filtering at write time — run the same instruction-detection checks used on input at the point of writing to memory, since an injected instruction could just as easily be phrased as a "fact to remember."
• Provenance tagging — tag every stored memory with where it came from (which user, which session, verified vs. unverified), so retrieval can weight or exclude low-trust sources.
• Anomaly monitoring — flag unusual write patterns, e.g. a single session trying to write many "facts" in a row, or facts that contradict previously verified ones.

This is genuinely tricky because embeddings aren't human-readable text you can grep for — you need to track, at write time, which raw records produced which vector IDs, so a deletion request can be translated into "delete these specific vector IDs" rather than trying to reverse-engineer embeddings later. In practice this means maintaining a mapping table (user ID → memory/vector IDs) alongside the vector store itself.

This is the classic cache invalidation problem, and an agent system actually has two very different kinds of cache to think about, each needing a different staleness strategy:

• LLM response cache — caching the model's answer for a semantically identical request (e.g. "what's your refund policy" asked by many users). This is safe to cache aggressively because the underlying answer rarely changes minute to minute — use a time-based TTL (say, 24 hours) matched to how often the source content actually changes, and invalidate immediately when the source document is updated by hooking into that update event rather than waiting for the TTL to expire.
• Tool-result cache — caching the output of an external API call (weather, stock price, order status). Here staleness is much more dangerous because the data is genuinely time-sensitive. The fix is to set the TTL to match the real-world volatility of the data, not one global default: weather might get a 15-minute TTL, a company's return policy might get a 24-hour TTL, and something like "current account balance" probably shouldn't be cached at all, or only for a few seconds.

Beyond TTL, there are three complementary invalidation strategies:

• Event-driven invalidation — when the underlying data changes (a price update, a policy edit), actively push a "delete this cache key" event instead of waiting for it to expire on its own. This is more accurate but requires the source system to emit change events.
• Write-through — whenever your own system updates the underlying data, update the cache at the same time, so it's never stale in the first place.
• Versioned cache keys — include a version or last-modified timestamp of the source data in the cache key itself, so a stale key naturally stops matching once the source changes, without needing an explicit delete.

You also want to be honest about what's cacheable at all — anything that must be correct in real time (payments, account balances, safety-critical data) should bypass the cache entirely rather than trying to tune a TTL tightly enough to feel safe.

Simple exact-match caching (hash the literal input string, look up the exact same string) only helps when users ask the identical thing verbatim, which is rare in natural language — "what are your hours" and "when are you open" are different strings but the same underlying request. For a meaningful cache hit rate, many teams use semantic caching instead: embed the incoming request, and check the cache for any previous request whose embedding is close enough (above a similarity threshold) to reuse its cached answer, rather than requiring an exact string match.

The tradeoff is risk of a false-positive hit — two questions that are similar in wording but actually need different answers ("what's the return policy for electronics" vs "for clothing"). This is managed by setting a conservative similarity threshold and by scoping the cache to a specific tool or intent category rather than caching globally.

Token cost is the most visible line item, but it multiplies with every loop iteration — a 4-step agent task can involve 4+ separate model calls for a single user request, not one. Beyond that, real cost centers include: vector store storage and query costs (embeddings at scale aren't free), the per-invocation cost of whatever compute runs your tools (Lambda invocations, container time), data transfer, and — often overlooked — the human review time for any approval-gated actions, which is a real operational cost even though it's not a cloud bill line item.

Put hard ceilings in the code layer, not just monitoring after the fact: a maximum number of loop iterations per task, a maximum token budget per request, per-user rate limits, and a circuit breaker that pauses a specific workflow if its cost in a time window crosses a threshold. Monitoring and alerting are necessary but not sufficient on their own — by the time a human sees an alert, a loop can have already burned through a large budget in seconds.

Latency compounds from every sequential round trip: each LLM reasoning step and each tool call is a network hop, and a multi-step task chains several of them one after another. The fix isn't "make the model faster" (though smaller models do help) — it's restructuring the workflow:

• Parallelize independent steps — if tool B doesn't depend on tool A's output, fire them concurrently instead of waiting for A to finish.
• Stream partial output — send status updates and partial text to the user as they're produced, so perceived latency drops even if total latency doesn't change.
• Route to a smaller/faster model for steps that don't need the largest model's reasoning depth (e.g. a routing decision itself can often use a fast, cheap model).
• Prefetch predictable data — if you know a step will likely need certain memory or tool data, kick off that fetch before the LLM even finishes deciding it needs it.

Keep each agent instance stateless — no per-instance in-memory session data — so any instance behind the load balancer can handle any request, and back all instances with one shared, external state store. This lets you scale horizontally by adding more identical instances rather than needing bigger ones, and it means a crashed instance doesn't lose anyone's session.

The vector store itself. LLM capacity and compute scale fairly linearly by adding more provisioned throughput or more instances, but a poorly indexed or unsharded vector database can become the slow point once you have millions of stored embeddings being searched per request — and it's the kind of bottleneck that doesn't show up until you're well past initial load testing with a small dataset.

First, distinguish transient failures (network blip, momentary rate limit) from real failures (the API is genuinely down, or your request is malformed). For transient failures, retry with exponential backoff and jitter — wait a bit longer each retry, with some randomness so many concurrent requests don't all retry at the exact same moment and hammer the recovering service simultaneously. Cap retries at 2-3 attempts; beyond that you're just adding latency for a call that's not going to succeed. If a dependency keeps failing across many requests, a circuit breaker should trip and stop sending traffic to it for a cooldown window, falling straight to a fallback path — this protects both your system's latency budget and the struggling downstream service.

Just as important: idempotency. If a tool call times out, you often don't know if it actually succeeded on the other end before the timeout. Retrying a non-idempotent action (like a payment) could double-execute it. The fix is to require an idempotency key on any mutating tool call, so retrying the same logical request is safe even if the first attempt's response was lost.

You need distributed tracing tied together by one correlation ID per user request/session, capturing: the exact prompt sent to the LLM (including which memory chunks were retrieved and which tool schemas were included), the model's raw response, every tool call with its parameters and result, and every memory read/write — each with a timestamp. Without this, you're guessing; with it, you can replay the entire decision chain and pinpoint exactly which input caused the wrong output.

Leading indicators, not just "is it up": tool-call error rate trending up, average steps-per-task creeping up (a sign of looping or confusion), retrieval relevance scores dropping, human-rejected-approval rate rising, and p95/p99 latency (not just average — tail latency is what users actually feel). Set alerting thresholds on the trend, not just a static value, since a slow creep is often the real warning sign, not a sudden spike.

Three concrete benefits: accuracy — a sub-agent with 5 relevant tools and a focused prompt makes far fewer tool-selection mistakes than one agent choosing among 40 tools across unrelated domains; security — least privilege is naturally easier to enforce per sub-agent, since a payments sub-agent's blast radius is contained to payment tools; and ownership — different teams can independently build, test, and deploy their own sub-agent without touching a shared monolith prompt that everyone is afraid to change.

The cost is coordination complexity and extra latency (each delegation is another round trip), so it's a genuine tradeoff, not a strict upgrade over a single agent.

Sub-agents can produce results that individually look correct but conflict with each other (a scheduling mismatch between two independently-booked services), the orchestrator can misroute a task to the wrong specialist, and debugging gets harder because a wrong final answer could originate from the orchestrator's routing decision, a sub-agent's tool call, or the merge step — you need tracing that spans across all of them with a shared correlation ID, otherwise a bug can hide at the "seam" between two agents that neither team notices in isolation.

Build a golden evaluation set — a representative sample of real tasks with known-correct tool choices and outcomes — and run the agent against it automatically on every change, scoring tool-selection accuracy, task completion rate, safety-policy violations, latency, and cost. Treat this exactly like a regression suite for traditional software: if a prompt or tool-schema change drops the eval score, you catch it before it reaches production traffic, not after a customer complains.

Beyond the automated eval set, adversarial testing matters specifically for agents: deliberately try prompt-injection-style inputs, ambiguous requests, and edge-case tool failures against the eval set to make sure the guardrails you built actually hold up, rather than just testing the happy path.

Stage it: ship to a small percentage of traffic first (say 5%), compare key metrics against the previous version side-by-side (completion rate, error rate, cost, safety flags), and only ramp up once you're confident there's no regression. Keep a fast, one-step rollback path — if you can't revert within minutes, the staged rollout doesn't actually protect you.