Practical Guide: Detecting and Automatically Blocking Anomalous Traffic

Why automated anomaly detection matters

Modern web services face noisy, high-volume traffic that can hide credential stuffing, layer-7 DDoS, or API abuse. Detecting anomalies quickly and converting detections into enforcement (rate-limits, IP blocks, captchas) reduces downstream outages and costs. This guide focuses on pragmatic, production-ready patterns you can implement with logs, metrics, and a simple ML or statistical detector—then enforce decisions through your CDN/WAF or on-prem firewall.

Architecture overview

• Telemetry: logs, metrics, and real-time request streams (e.g., access logs, HTTP headers, request rates).
• Detection engine: rules, statistical detectors, or unsupervised ML models operating on sliding windows.
• Enrichment & scoring: geo/IP reputation, known proxies, user-agent heuristics.
• Decisioning: apply thresholds, confidence windows, and alerting policies to avoid false positives.
• Enforcement: CDN/WAF APIs, load-balancer rules, iptables/nftables on hosts, or application-layer mitigations.

Data sources and features

• Request count per client IP, per API key, per endpoint.
• Request rate distributions (per minute / per second).
• Error-rate anomalies (5xx spikes), latency spikes.
• Behavior features: unique endpoints accessed, header entropy, cookie presence.

Detection methods — quick comparison

• Static thresholds: Simple and low-latency. Good for obvious rate limits, but brittle and high-maintenance.
• Statistical (z-score, EWMA): Fast, explainable, adapts to seasonality with rolling windows. Works well for traffic-volume anomalies.
• Unsupervised ML (isolation forest, clustering): Detects subtle multi-dimensional anomalies but needs calibration and periodic retraining.
• Supervised models: Effective if you have labeled attacks, but requires high-quality labels and incurs maintenance.

Simple production-ready pipeline (pattern)

1. Aggregate streaming logs into per-key metrics using a streaming job (Kinesis/Firehose, Pub/Sub, Kafka + stream processors).
2. Compute short-window metrics (1 min, 5 min) and baseline metrics (hourly/daily) for seasonality.
3. Run lightweight detectors (EWMA + z-score) to flag high-confidence anomalies.
4. Enrich flagged events (geo, ASN, recent history) and compute a final score.
5. Trigger enforcement with rate-limited API calls to the enforcement layer; escalate to human review if confidence is low.

Python example: rolling z-score detector and enforcement hooks

The example below shows a minimal detector that ingests per-minute counts for an IP, computes a rolling mean/std, and triggers enforcement when the z-score is large. Replace the enforce_block implementation with your CDN/WAF API or host firewall command.

import collections
import math
import time
import requests
 
WINDOW = 60  # keep last 60 minutes
THRESHOLD = 6.0  # z-score threshold for strong anomalies
 
counts = collections.deque(maxlen=WINDOW)
 
def rolling_stats(deque_counts):
    if not deque_counts:
        return 0.0, 0.0
    n = len(deque_counts)
    mean = sum(deque_counts) / n
    var = sum((x - mean) ** 2 for x in deque_counts) / n
    std = math.sqrt(var)
    return mean, std
 
def enforce_block(ip, reason):
    # Example: call your WAF/CDN API or run host firewall commands here.
    # Keep enforcement idempotent and rate-limited.
    print(f"ENFORCING block {ip} reason={reason}")
    # Example placeholder: HTTP call to your enforcement endpoint
    # requests.post('https://waf.example/api/block', json={'ip': ip, 'reason': reason}, headers={'Authorization': 'Bearer TOKEN'})
 
# Simulated per-minute ingestion loop for one IP
while True:
    # Replace this with real ingestion from metrics store
    current_minute_count = int(input('requests this minute: '))
    counts.append(current_minute_count)
 
    mean, std = rolling_stats(counts)
    if std == 0:
        score = 0.0
    else:
        score = (current_minute_count - mean) / std
 
    print(f"count={current_minute_count} mean={mean:.2f} std={std:.2f} z={score:.2f}")
 
    if score > THRESHOLD and len(counts) > 10:
        # additional enrichment checks could go here
        enforce_block('198.51.100.23', f'zscore={score:.2f}')
 
    # sleep or wait for next minute batch
    time.sleep(0.1)

Notes

• Operate detectors per key (IP, API key, user ID) and keep metrics in a time-series DB or in-memory store (Redis) for low latency.
• Persist detection events and enforcement actions for audit and rollback.

Enforcement examples

Enforcement should be incremental and reversible. Start with milder actions (challenge/captcha, rate-limit) and escalate to IP blocks when confidence is high.

Host-level (iptables) example

On a host you control, you can add a temporary iptables DROP rule. Make sure to use a centralized mechanism to avoid rule drift across a fleet.

import subprocess
 
def block_ip_host(ip, duration_seconds=3600):
    # Add a DROP rule (example for Linux iptables). Running as root required.
    subprocess.run(['iptables', '-I', 'INPUT', '-s', ip, '-j', 'DROP'], check=True)
    # Schedule removal after duration_seconds in your orchestrator / cleanup job.
 
# block_ip_host('198.51.100.23')

CDN/WAF API (example pattern)

Most CDNs and WAFs provide an API to create rules or IP lists. Use an allowlist/denylist pattern and avoid creating too many rules—use dynamic lists where possible. Below is a generic blocking API call pattern; replace the URL and payload per your provider.

import requests
 
WAF_API = 'https://waf.example/api/v1/blocks'  # replace with provider endpoint
API_TOKEN = 'REPLACE_WITH_TOKEN'
 
def block_ip_waf(ip, comment='auto-detected'):
    headers = {
        'Authorization': f'Bearer {API_TOKEN}',
        'Content-Type': 'application/json',
    }
    payload = {'ip': ip, 'comment': comment, 'ttl_seconds': 3600}
    resp = requests.post(WAF_API, json=payload, headers=headers, timeout=5)
    resp.raise_for_status()
    return resp.json()
 
# block_ip_waf('198.51.100.23')

Streaming detector example (Node.js) with EWMA

EWMA reacts faster than simple moving average and is cheap to compute per-key. This Node.js snippet demonstrates a per-IP EWMA-based anomaly score suitable for high-throughput streams.

const EWMA_ALPHA = 0.3;
const THRESHOLD = 8.0;
 
class EwmaTracker {
  constructor(alpha = EWMA_ALPHA) {
    this.alpha = alpha;
    this.ewma = 0;
    this.ewmVar = 0;
    this.count = 0;
  }
 
  update(value) {
    if (this.count === 0) {
      this.ewma = value;
      this.ewmVar = 0;
    } else {
      const delta = value - this.ewma;
      this.ewma += this.alpha * delta;
      this.ewmVar = (1 - this.alpha) * (this.ewmVar + this.alpha * delta * delta);
    }
    this.count++;
  }
 
  score(value) {
    const std = Math.sqrt(this.ewmVar || 1e-6);
    return (value - this.ewma) / std;
  }
}
 
// Usage in a stream processor: maintain a map of trackers per IP and feed per-minute counts.

Operational tradeoffs

• False positives: Costly if you block legitimate users. Mitigate with multi-step enforcement, confidence windows, and human review for high-impact blocks.
• Latency vs accuracy: Faster detectors use fewer features and shorter windows; ML models may be slower but more accurate.
• Scale: Keep per-key state in a scalable store (Redis, in-memory with sharding). For very high cardinality, pre-aggregate by subnet or user class.
• Cost: Calling external WAF/CDN APIs at high rates may be rate-limited or billed—batch updates and use dynamic blocklists when supported.

Deployment and safety best practices

• Start with monitoring-only mode to collect metrics and evaluate precision/recall before enforcing.
• Implement an exponential backoff / cooldown for enforcement—avoid thrashing rules on noisy signals.
• Keep a rollback channel: maintain an allowlist or a way to reverse blocks automatically if false-positive patterns are detected.
• Audit logs: record detector inputs, scores, and enforcement actions for compliance and debugging.

Conclusion

Automated anomaly detection tied to enforcement reduces operational load and improves resilience, but must be implemented with careful enrichment, confidence thresholds, and reversible enforcement. Start with simple statistical detectors (EWMA, z-score) running in monitoring mode, validate with historical data, then gradually enable automated enforcement with incremental actions. Use centralized state stores, keep audit logs, and tune policies to minimize false positives.

Further reading: the original writeup that inspired this approach is available on DEV Community.