Practical Guide: Self-host LLMs on Small Droplets with vLLM — Deploy Mistral, Llama, and Phi-3

Introduction

Over the last year, multiple guides showed you can run modern open-source LLMs on low-cost cloud droplets. This article condenses practical, production-oriented advice: a minimal architecture using vLLM, a Docker Compose example you can adapt, a Python client to call your model, operational tradeoffs, and hardening tips. The goal is code-first: get a reliable endpoint in hours and understand the recurring costs and maintenance.

Why use vLLM on a droplet?

• Fast startup and batching: vLLM is optimized for efficient CPU/GPU memory use and request batching.
• Lower cost: Smaller droplets or low-cost GPUs let you run 7B-class models for development and some production workloads.
• Control and privacy: Full control over data, model weights, and inference latency.

Minimal architecture

At a minimum you need:

1. Model server (vLLM or equivalent) that loads the model and exposes a HTTP/gRPC endpoint.
2. Reverse proxy (optional) for TLS, rate limiting, and basic auth.
3. Persistent storage for model weights (NFS or attached block storage).
4. Monitoring and logging (Prometheus, Grafana, Loki or a lightweight alternative).

When to pick CPU vs GPU

• GPU: Required for low-latency, high-throughput inference on models >7B (or quantized variants). Use small GPUs (e.g., A10, T4) if available on your provider.
• CPU: Feasible for tiny or heavily quantized models (e.g., Phi-3 Mini, 4-6B quantized) but expect higher latency.

Docker Compose example (quick start)

This Compose sets up vLLM behind Caddy as a TLS reverse proxy. Adapt volumes and model path to your storage. It assumes you already have model weights in /models/mistral-7b.

version: "3.8"
services:
  vllm:
    image: ghcr.io/vllm/vllm:latest
    command: ["vllm-server", "--model", "/models/mistral-7b", "--port", "8000", "--num-gpus", "1"]
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    volumes:
      - /models:/models:ro
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
    restart: unless-stopped
 
  caddy:
    image: caddy:latest
    ports:
      - 443:443
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
      - caddy_config:/config
    depends_on:
      - vllm
    restart: unless-stopped
 
volumes:
  caddy_data:
  caddy_config:

Notes:

• Replace the vLLM image and command with the official vLLM CLI you prefer. See the vLLM repo for up-to-date flags: vLLM on GitHub.
• On CPU-only droplets, remove the GPU-related lines and set num-gpus to 0; expect much higher latencies.

Basic Caddyfile (TLS & reverse proxy)

your.example.com {
  reverse_proxy vllm:8000
  encode gzip
  header {
    Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
  }
}

Testing the endpoint

Use a small Python client to call the inference endpoint. The snippet below assumes your vLLM server exposes a JSON REST API compatible with /generate style endpoints; adapt to your server's endpoint.

import requests
 
url = "https://your.example.com/generate"
headers = {"Content-Type": "application/json"}
payload = {
    "prompt": "Write a short checklist for deploying models on droplets:",
    "max_tokens": 120,
}
resp = requests.post(url, json=payload, headers=headers, timeout=30)
resp.raise_for_status()
print(resp.json())

Operational guidance

• Model downloads: Download model weights to attached block storage ahead of time to avoid long container startup times.
• Swap and memory: For CPU droplets, configure swap and monitor OOMs. Swapping increases latency but avoids crashes for oversized models.
• Auto-scaling: For production, front your model with a lightweight autoscaler (Kubernetes + KEDA, or horizontal autoscaling with a queue) and use batching at the model server.
• Quantization: Use 4-bit or 8-bit quantized weights where supported to reduce memory pressure and cost. Verify accuracy for your tasks.

Security and licensing

1. Confirm the model's license permits your intended use (commercial vs research). Different LLM families have different conditions.
2. Run your model in a VPC, expose only the proxy endpoint, and enable TLS and authentication for any public endpoints.
3. Sanitize prompts and outputs if you process user data to reduce exfiltration risk.

Tradeoffs

• Cost vs performance: Small droplets are cheap but may not meet latency SLAs. GPUs increase cost but improve latency and throughput.
• Maintenance: Self-hosting requires patching, monitoring, and handling model upgrades. Managed APIs offload that work at the cost of control and recurring fees.
• Model updates: Upgrading weights or switching quantization can be disruptive; automate blue/green deployments to reduce downtime.

Checklist: Launch in a few hours

1. Choose model and check license.
2. Pick droplet with enough RAM or GPU fraction for your model.
3. Pre-download model weights to attached storage.
4. Deploy with Compose or container runtime; add a TLS proxy.
5. Test with a local client and enable monitoring and alerting.

Further reading and tools

See practical deployment walkthroughs for reference and ideas on low-cost provider setups: Mistral 7B + vLLM on a low-cost droplet and the vLLM project page linked earlier.

Conclusion

Self-hosting modern LLMs on small cloud droplets is now feasible for many developer workflows. Use vLLM or similar servers for efficient batching, prefer GPUs for production low-latency needs, and automate model storage and deployment. Balance cost, performance, and maintenance: for sensitive data or full control, self-hosting is compelling; for scale and simplicity, managed inference may still be better. Start small, measure latency and cost, and iterate from there.

Tip: Keep a lightweight staging droplet with identical software to validate model upgrades before rolling to production.