Vector DB Evals in 2026: Weaviate vs pgvector Recall Test

You shipped RAG to production in Q4, and now support is escalating tickets where the answer was sitting in the corpus the whole time. You run a quick sanity check: your top-5 recall looks fine on the twelve queries you tried by hand, and the vendor benchmark you cited during the architecture review claims 99% recall at sub-10ms. Both are true and neither predicts what happens on your data. Meanwhile the pgvector-vs-dedicated-store debate has restarted in your team channel for the third time in six months, and nobody can settle it because nobody has numbers from your corpus, your chunk sizes, your query distribution — just blog posts benchmarking ANN-Benchmarks SIFT vectors that resemble your embeddings not at all.

This is for developers who already have embeddings in a database and need to defend a retrieval decision with evidence. You should be comfortable with Docker Compose, Python, SQL, and reading a query plan; you should know what an embedding is and roughly how HNSW differs from IVF. It does not teach you to build a RAG app, fine-tune an embedding model, or pick an LLM — the generation half of the stack is out of scope. It is also not a verdict piece: the point is a harness that produces your answer, not ours.

AI is genuinely useful in the tedious middle of this work — drafting synthetic queries against your documents, expanding a seed set of golden questions, writing the boilerplate around metric computation, and summarizing where two runs diverged. It is unreliable at the part that decides everything: judging whether a retrieved chunk actually answers the question. LLM-as-judge relevance labels drift with prompt wording, favor fluent-but-wrong passages, and quietly inflate nDCG in ways that survive every automated check you write. Human review of the golden set is non-negotiable — if a domain expert has not eyeballed your relevance labels, your benchmark is measuring your judge prompt, not your retriever.

What This Guide Covers

• How to read published vector database benchmarks critically, and which 2026 changes to both engines actually invalidate older comparisons
• A working mental model of how Weaviate and pgvector differ architecturally — and which differences show up in your latency budget versus which are marketing
• A repeatable method for building a test corpus and golden question set that reflects real user queries instead of the ones you thought of on a Tuesday
• Which retrieval metrics to trust for which decision, why recall@k alone will mislead you, and how to report results your team can argue with productively
• Reproducible local environments for both stacks so a colleague can rerun your numbers instead of taking your word for them
• The design of an evaluation harness that produces stable, comparable results across runs and across engines
• How the tuning knobs on each index actually trade recall against latency and memory — and how to find your operating point instead of copying someone else’s config
• Why chunking strategy frequently swamps index choice entirely, and how to run ablations that expose this before you migrate anything
• How to combine keyword and dense retrieval, and how fusion behaves when your two retrievers disagree
• Where rerankers genuinely earn their latency, how two leading options compare on cost and quality, and when stacking one is cheaper than upgrading your index
• Honest cost modeling per million vectors — memory footprint, query latency under concurrency, and what actually scales badly
• How both engines behave on inserts, deletes, and rebuilds while serving traffic, which is where production retrieval quality usually degrades
• The evaluation mistakes that silently corrupt results — leakage, mismatched normalization, stale indexes, and metric definitions that don’t mean what you assume
• How to wire retrieval quality into CI so a chunking change or model swap fails the pull request instead of surfacing in support tickets
• A decision framework for migrate, stay, or run both — with case studies showing what each choice cost the teams that made it

Instant online access the moment checkout completes — no waiting on an email, no drip schedule, no upsell sequence. You buy it, you read it.