Deploy Llama 3.2 with Ollama + PostgreSQL Vector Caching — Budget RAG Guide

Why this pattern and when to use it

Running a local model server (Ollama) alongside a lightweight vector cache (PostgreSQL + pgvector) gives you low-latency inference, reproducible model versions, and fine-grained control of data. This setup is ideal when you want to: reduce API costs, keep data on-premise, or build a deterministic Retrieval-Augmented Generation (RAG) service with predictable performance.

High-level architecture

Ollama runs the LLM locally and serves inference/embeddings via HTTP or CLI.
PostgreSQL + pgvector stores document vectors, metadata, and acts as a cache to avoid recomputing embeddings and repeated model calls.
App layer (Node/Python/Go) orchestrates: check cache & vector search → call model for missing embeddings or generation → update cache.

Prerequisites & tradeoffs

Budget droplets have limited RAM/CPU: prefer quantized or smaller Llama variants for low-memory inference, or rely on model-resident caching to avoid repeated loads.
Security: expose Ollama and Postgres only to your application or a private network; enable TLS and authentication for production.
Durability: Postgres persists vectors; snapshot/backup volumes regularly.

Quick plan (what you’ll build)

Provision a low-cost droplet and install Docker.
Install Ollama (model server) on the droplet.
Run a Postgres container that includes the pgvector extension (Dockerfile + docker-compose).
Create a vector table and implement a simple cache+search workflow in your app.

Step 1 — Prepare the droplet

Update packages and install Docker and common tools.

sudo apt-get update && sudo apt-get upgrade -y
sudo apt-get install -y docker.io docker-compose git curl build-essential

Install Ollama following the official instructions. Use the vendor docs for the latest installer and security notes:

https://ollama.com/ (official install & docs)

Step 2 — Postgres + pgvector in Docker

Building a Postgres image that compiles and installs the pgvector extension at container build time is a robust way to get vector support on a small droplet.

Example Dockerfile (place in pgvector-postgres/Dockerfile):

FROM postgres:15
 
# Install build dependencies and compile pgvector from source
RUN apt-get update \
  && apt-get install -y --no-install-recommends \
    build-essential git postgresql-server-dev-15 \
  && git clone https://github.com/pgvector/pgvector.git /tmp/pgvector \
  && cd /tmp/pgvector \
  && make \
  && make install \
  && rm -rf /tmp/pgvector \
  && apt-get remove -y build-essential git postgresql-server-dev-15 \
  && apt-get autoremove -y \
  && rm -rf /var/lib/apt/lists/*

Compose file (minimal) in the same folder as the Dockerfile:

version: '3.8'
services:
  db:
    build: ./pgvector-postgres
    environment:
      POSTGRES_PASSWORD: examplepassword
      POSTGRES_DB: appdb
    volumes:
      - pgdata:/var/lib/postgresql/data
    ports:
      - "5432:5432"
 
volumes:
  pgdata:

Bring up the database:

docker-compose up -d --build

Step 3 — Initialize the vector table

Connect with psql and create the extension and table. Adjust the vector dimension to match your model's embedding size (e.g. 1536).

psql -h localhost -U postgres -d appdb
 
-- In psql:
CREATE EXTENSION IF NOT EXISTS vector;
 
CREATE TABLE documents (
  id SERIAL PRIMARY KEY,
  content TEXT NOT NULL,
  embedding VECTOR(1536),
  metadata JSONB
);
 
CREATE INDEX ON documents USING ivfflat (embedding vector_l2_ops) WITH (lists = 100);

Step 4 — Application logic: cache, embed, search

The flow is:

Check if a document has an embedding in Postgres.
If missing, call your local Ollama server to compute the embedding and INSERT it.
Perform nearest-neighbor search with the cached embeddings to build context for generation.
Call the model for the final generation (RAG).

Example Python snippet (illustrative):

import requests
import psycopg2
import json
 
# Configuration
DB_DSN = "dbname=appdb user=postgres password=examplepassword host=localhost"
MODEL_EMBED_ENDPOINT = "http://127.0.0.1:11434/embeddings"  # adjust to your model server
EMBED_DIM = 1536
 
# Helpers
conn = psycopg2.connect(DB_DSN)
 
def get_cached_embedding(text):
    with conn.cursor() as cur:
        cur.execute("SELECT id, embedding FROM documents WHERE content = %s", (text,))
        row = cur.fetchone()
        return row
 
def upsert_embedding(text, embedding, metadata=None):
    with conn.cursor() as cur:
        cur.execute(
            "INSERT INTO documents (content, embedding, metadata) VALUES (%s, %s, %s) ON CONFLICT DO NOTHING",
            (text, embedding, json.dumps(metadata or {}))
        )
        conn.commit()
 
def compute_embedding_local(text):
    # Example: call your local model server. Adjust payload/URL to match your server's API.
    payload = {"model": "llama-3.2", "input": text}
    r = requests.post(MODEL_EMBED_ENDPOINT, json=payload, timeout=30)
    r.raise_for_status()
    return r.json()["data"][0]["embedding"]
 
# Usage
text = "Example document to store and search"
cached = get_cached_embedding(text)
if not cached:
    embedding = compute_embedding_local(text)
    upsert_embedding(text, embedding, metadata={"source": "import"})
else:
    print("Cache hit, id=", cached[0])
 
# Vector search example: find top 5 nearest docs to a query embedding
query_embedding = compute_embedding_local("Search query here")
with conn.cursor() as cur:
    cur.execute(
        "SELECT id, content, metadata, embedding <-> %s AS distance FROM documents ORDER BY distance LIMIT 5",
        (query_embedding,)
    )
    results = cur.fetchall()
    for r in results:
        print(r)

Notes:

The example assumes your local model server exposes an embeddings endpoint; consult your model server docs and adapt the request/response shapes.
When storing vectors, use the correct array/byte type mapping for your DB driver; psycopg2 handles Python lists into pgvector if you register the adapter, or send as a PostgreSQL array literal.

Operational tips & tradeoffs

Memory vs. latency: Smaller/quantized models reduce memory pressure but may increase latency or reduce quality. Use model caching (keep the model loaded) to reduce repeated cold-starts.
Indexing: IVFFlat or HNSW indexes accelerate search at the cost of additional memory and an index build step. Tune lists or HNSW parameters for your dataset and hardware.
Cache policy: Use LRU or TTL on your application side for seldom-used embeddings; store long-lived document vectors persistently in Postgres to avoid recompute.
Security: Bind services to localhost or private networks, enable firewall rules, and do not expose Ollama or Postgres directly to the public internet.

Scaling guidance

Start with a single droplet for dev — move Postgres to a dedicated instance if your dataset grows to reduce IO contention.
For heavy inference, separate model-serving nodes and use a lightweight request router to load-balance model calls.
Use object storage (S3/MinIO) to store large documents and only index smaller text chunks in Postgres.

Monitoring and observability

Track cache hit rate, embedding compute time, and vector search latency.
Log model versions and seed your RAG prompt with model metadata so outputs are reproducible.

Conclusion

This pattern—local Ollama model server + Postgres/pgvector cache—gives a practical balance of cost, control, and latency for production RAG on modest cloud hardware. The most important engineering choices are selecting a model that fits your RAM, tuning your vector index, and implementing a sensible caching policy to avoid repeated embedding work. Start small, measure cache hit rates and latency, then iterate on model size and index configuration.

How to Deploy Llama 3.2 with Ollama + PostgreSQL Vector Caching on a Budget Droplet — Practical RAG Guide