Production Patterns for AI Agents — State, Orchestration, Testing, Safety

Why this matters

AI agents (multi-step LLM-driven programs that plan, act, and persist state) are rapidly moving from experiments into user-facing systems. Many teams hit the same deployment wall: brittle behavior, runaway costs, poor observability, and security issues. This article gives practical, implementable patterns you can apply today to make agents reliable, testable, and maintainable in production.

High-level patterns

Choose combinations of these patterns to match your product constraints. There are no one-size-fits-all answers; tradeoffs are explicit below.

1) Stateful agents vs stateless loops

Decision:

Stateless — simple, scales with request load, best for single-turn reasoning or when external system stores authoritative state.
Persistent (stateful) — required when conversation context, memory, or long-running workflows are core; needs a durable store and GC strategy.

2) Orchestration & isolation

Run agents in a small, predictable execution environment with job queues, timeouts, and capability-limited workers. Use a central orchestrator to manage retries, backoffs, and handoffs between specialized agents.

3) Hybrid model strategy

Mix cheaper open-source models for background planning and lightweight tasks with higher-quality commercial models for final responses or safety-critical steps. Use a cost-aware router to decide model selection per step.

4) Observability and testing

Capture structured traces of prompts, tool calls, decisions, and state diffs. Treat each agent run as a traceable transaction. Build a test harness that can replay deterministic steps with mocked model outputs.

Concrete architecture

API / UI receives user input and enqueues a job.
Worker pool picks jobs from a durable queue (Redis, SQS, RabbitMQ) and materializes agent state from a store (Redis, DynamoDB, Postgres).
Agent executes a bounded loop: plan > call model > call tool (external API) > update state > emit telemetry.
Orchestrator enforces runtime limits, model selection rules, and circuit breakers for external tools.
Observability pipeline stores traces and metrics (logs, events, ML metrics) for debugging and audits.

Actionable code examples

Below are small, practical snippets illustrating the main ideas. They are illustrative; adapt to your stack.

Persistent agent loop (simplified)

<?php
// Simplified persistent agent loop using Redis for queue + state.
// Replace call_llm(), call_tool(), and serialize/deserialize with your stack.
$redis = new Redis();
$redis->connect('127.0.0.1', 6379);
 
while (true) {
    $raw = $redis->lPop('agent:queue');
    if (!$raw) { sleep(1); continue; }
 
    $task = json_decode($raw, true);
    $stateKey = "agent:state:" . $task['conversation_id'];
    $stateJson = $redis->get($stateKey) ?: '{}';
    $state = json_decode($stateJson, true);
 
    // Bounded loop: limit steps per job to avoid runaway compute
    for ($step = 0; $step < 5; $step++) {
        $prompt = build_prompt($task['input'], $state);
        $llmResponse = call_llm($prompt, ['model' => $task['model']]);
 
        // Example: agent plans a tool call
        if (isset($llmResponse['tool_call'])) {
            $toolResult = call_tool($llmResponse['tool_call']);
            $state['last_tool_result'] = $toolResult;
        } else {
            $state['last_response'] = $llmResponse['text'] ?? '';
            break; // done
        }
    }
 
    $redis->set($stateKey, json_encode($state));
}
?>

Model router (cost-aware, hybrid)

// Very small router to choose model by step type
function choose_model(array $step): string {
    // low-cost planning model for thoughts; high-quality for final answers
    if ($step['type'] === 'planning') return 'open-llm-small';
    if ($step['type'] === 'safety-check') return 'commercial-qa-xl';
    return 'open-llm-medium';
}
 
// Example usage
$step = ['type' => 'planning'];
$model = choose_model($step);

Test harness: mocking LLMs for deterministic tests

// Replace network call with a deterministic mock in tests
function call_llm($prompt, $opts = []) {
    if (defined('TEST_MODE') && TEST_MODE) {
        // Use a map of prompt hashes to canned replies
        $map = json_decode(file_get_contents(__DIR__ . '/fixtures/llm_map.json'), true);
        $key = md5($prompt);
        return $map[$key] ?? ['text' => 'mock default'];
    }
 
    // Production: make real API call
    return real_llm_api_call($prompt, $opts);
}
 
// In tests: record a trace once, then assert agent state evolves as expected
define('TEST_MODE', true);

Operational concerns & tradeoffs

Latency vs cost: Using cheaper models reduces cost but can increase error rate. Use async background steps to keep UX snappy while expensive verification runs later.
State store choice: Redis and DynamoDB are great for small JSON state and fast access. Use Postgres when you need complex queries, joins, or ACID guarantees.
Security: Principle of least privilege for agents that can call tools. Sandbox any tool that executes code or accesses secrets. Limit agent capabilities by role and job type.
Testing: Mock models and external tools. Record and replay traces to make integration tests deterministic.
Observability: Store prompts (or digests), decisions, and state diffs. Privacy: redact sensitive fields or store only hashes where necessary to comply with regulations.

Quick checklist for deployment

Limit steps per run and enforce wall-clock timeouts.
Use durable queue + state store; design GC for stale conversations.
Implement model routing with cost caps and fallbacks.
Add structured tracing of prompt & tool calls for audits.
Build test harnesses with deterministic mocks and replay capability.
Apply least-privilege and sandboxing for tool execution.

Conclusion

Deploying AI agents reliably requires thinking beyond prompts: durable state, bounded execution, observability, and cost-aware model routing. Start with a small, sandboxed worker that persists state and add tracing and testing early. The patterns above give a practical, incremental path from prototype to production.

Next steps

Implement a small persistent worker using the queue + Redis pattern and add a test harness with mocked LLM replies.
Add observability for a single agent workflow, then expand to cross-agent traces.
Introduce a model router and measure cost vs quality for typical flows before wide rollout.

Production Patterns for AI Agents: State, Orchestration, Testing, and Safety