Multi-model API Routing: Practical Patterns for Cost, Latency, and Capability-Based LLM Selection

Why multi-model routing matters

Using a single LLM vendor rarely fits every production need. New entrants (DeepSeek and others) and self-hosted models (Llama 2, local inference) change the calculus: lower cost providers, capability differences, latency tradeoffs, and privacy requirements push teams toward a routing layer that picks the best model per request. For an engineer-facing field report and practical experiences, see I Wish I Knew Multi-Model API Routing Sooner.

Goals for a production router

• Correctness: send prompts to a model with the right capabilities (code, summarization, retrieval, chat).
• Cost control: prefer cheaper models when acceptable for quality.
• Latency & SLOs: meet interactive latency budgets by avoiding high-latency backends for short requests.
• Resilience: fail over to alternate models on error or rate-limit events.
• Privacy/residency: support routing to on-prem or local LLMs for sensitive data.

Routing strategies (high level)

• Rule-based — deterministic rules based on intent, user flags, or prompt tags (fast, predictable).
• Cost-based — pick the model with the cheapest estimated price for the request subject to constraints.
• Latency-based — route to lower-latency deployments for interactive flows.
• Capability-based — choose higher-quality models for code or chain-of-thought tasks.
• Ensemble / Staging — run a fast, cheap model and validate / augment with a stronger model asynchronously.

Implementation: a practical router (pattern + code)

The sample below shows a compact JavaScript-style router you can adapt to your stack. The logic is simple: tag the prompt to derive intent, estimate tokens and cost, apply user constraints (max latency, prefer-local), choose the best candidate, call the provider, and fall back on error.

const MODELS = {
  openai: { id: 'openai-gpt-4o', costPerToken: 0.0003, latencyMs: 300, capabilities: ['chat','code','summary'] },
  deepseek: { id: 'deepseek-1', costPerToken: 0.00005, latencyMs: 400, capabilities: ['search','summary'] },
  local: { id: 'llama2-local', costPerToken: 0.0, latencyMs: 200, capabilities: ['chat','summary'], isLocal: true }
};
 
function estimateTokens(prompt) {
  // cheap heuristic (replace with a token-estimator for your tokenizer)
  return Math.max(1, Math.ceil(prompt.length / 4));
}
 
function detectIntent(prompt) {
  const p = prompt.toLowerCase();
  if (p.includes('write code') || p.includes('javascript') || p.includes('python')) return 'code';
  if (p.includes('summarize') || p.includes('tl;dr')) return 'summary';
  return 'chat';
}
 
function estimateCost(tokens, model) {
  return tokens * model.costPerToken;
}
 
function chooseModel({prompt, userPrefs = {}}) {
  const intent = detectIntent(prompt);
  const tokens = estimateTokens(prompt);
 
  // If user explicitly requires local inference
  if (userPrefs.preferLocal && MODELS.local) return MODELS.local;
 
  // Candidate filtering by capability
  const candidates = Object.values(MODELS).filter(m => m.capabilities.includes(intent));
 
  // Rank candidates by (cost + latency penalty) subject to maxLatency
  const maxLatency = userPrefs.maxLatency || 1000;
  const scored = candidates
    .map(m => ({ m, cost: estimateCost(tokens, m), latency: m.latencyMs }))
    .filter(x => x.latency <= maxLatency);
 
  if (scored.length === 0) return candidates[0] || MODELS.openai; // best-effort
 
  // Simple combined score: cost first, break ties by latency
  scored.sort((a, b) => (a.cost - b.cost) || (a.latency - b.latency));
  return scored[0].m;
}

This function is intentionally compact. In production:

• Replace estimateTokens with a tokenizer-aware estimator (BPE/spacy/OpenAI tokenizer).
• Keep a dynamic model registry (costs and latencies can change) stored in config or service discovery.
• Use feature flags to route subsets of traffic to new providers (canary experiments).

Example: Express-style endpoint with fallbacks

app.post('/api/generate', async (req, res) => {
  const { prompt, userPrefs } = req.body;
  const primary = chooseModel({ prompt, userPrefs });
 
  try {
    const result = await callModelApi(primary, prompt);
    return res.json({ model: primary.id, result });
  } catch (err) {
    // common failures: rate-limit, timeout, 5xx
    console.warn('primary model failed', primary.id, err.message);
 
    // fallback order: prefer local > cheapest alternative > strongest
    const alternatives = Object.values(MODELS).filter(m => m.id !== primary.id && m.capabilities.includes(detectIntent(prompt)));
    for (const alt of alternatives) {
      try {
        const result = await callModelApi(alt, prompt);
        return res.json({ model: alt.id, result, fallback: true });
      } catch (e) {
        console.warn('fallback failed', alt.id, e.message);
      }
    }
 
    // global failure
    return res.status(502).json({ error: 'All model backends failed' });
  }
});

Provider call abstraction

Keep provider-specific clients behind an adapter. That centralizes rate-limit handling, error classification (retryable vs permanent), and request signing.

async function callModelApi(model, prompt) {
  if (model.isLocal) {
    // call your local inference endpoint
    return await fetch('http://localhost:8000/infer', { method: 'POST', body: JSON.stringify({ model: model.id, prompt }) });
  }
 
  if (model.id.startsWith('openai')) {
    // call OpenAI / similar
    return await fetch('https://api.openai.example/v1/generate', { method: 'POST', headers: { 'Authorization': 'Bearer ...' }, body: JSON.stringify({ model: model.id, prompt }) });
  }
 
  if (model.id.startsWith('deepseek')) {
    // DeepSeek-like provider
    return await fetch('https://api.deepseek.example/v1/generate', { method: 'POST', body: JSON.stringify({ prompt }) });
  }
 
  throw new Error('Unknown model');
}

Operational considerations

• Observability: emit model selection, latency, tokens consumed, and provider error codes to metrics. Track cost-per-request in billing dashboards.
• Rate limits & retry policy: map provider errors to retry vs fail. Implement exponential backoff and circuit breakers per-provider.
• Privacy & caching: sensitive prompts may require local-only routing; cache public responses where safe to reduce costs.
• Testing & canaries: A/B route a small percentage to new providers and compare quality & hallucination rates before increased traffic.
• Token accounting: keep a running estimate of tokens used per model call and reconcile with provider billing periodically.

Tradeoffs to weigh

1. Simplicity vs accuracy — rule-based routers are fast to implement but can misroute edge cases. ML-based routers (intent classifiers) reduce misroutes at the cost of complexity.
2. Cost vs quality — cheaper providers save money but may underperform on complex tasks; consider hybrid patterns (cheap first, strong second).
3. Latency vs throughput — local models can be faster for short requests but might not scale for high throughput without investment in infra.
4. Operational overhead — more providers add more integration work: billing, credentials, monitoring, legal.

Actionable checklist to get started

• Instrument prompt types and token usage across current traffic to get baseline cost & latency.
• Implement a minimal router (3–5 rules) and an adapter interface for providers.
• Run a small canary: route 1–5% of traffic to an alternate provider and compare results and costs.
• Add fallbacks and circuit breakers: never expose end-users to raw provider outages.
• Automate periodic re-evaluation of provider costs and latencies (daily/weekly updates to the model registry).

Further reading and examples

For personal field notes about implementing routing in a production backend, see the field report. For practical tips when trying new, cost-optimized providers, the community posts about switching to DeepSeek can be helpful to understand integration and billing differences: DeepSeek cost notes.

Conclusion

Multi-model routing turns a single-vendor LLM integration into a flexible, cost-aware, and resilient service. Start with clear goals (latency SLOs, privacy constraints, cost thresholds), instrument aggressively, and iterate with canaries and fallbacks. The incremental complexity pays off when you need to balance cost, quality, and regulatory constraints in production.