Stop Your AI From Confidently Making Things Up: A Practical RAG Guide for Vibe Projects

Before You Write a Line of Code: Four Questions for Whether You Need RAG at All

Before building RAG Q&A, answer an uncomfortable question: do you actually need RAG? By 2026, RAG has become "the default answer" — but the default answer is not the right answer. Vibe projects run on scarce resources, and spending two weeks on a feature that 50 lines of code could have handled is the most expensive kind of waste.

Decide with four questions — the more "yes" answers, the more RAG is worth it:

Question	What a "yes" means
Is the data private?	The model never saw it in training, and you can't cram it into a prompt — only retrieval can reach it
Does the data change often?	Editing a document is far cheaper than fine-tuning a model; RAG is a natural fit
Must answers be traceable?	Users want to see "which document this sentence came from" — RAG ships with a citation chain
Is the Q&A open-domain?	Users will ask questions you never imagined; FAQ templates can't cover them

Three or four "yes" answers, and RAG is your thing. Two — proceed, but read the discouragement list below first. Zero to one — close this article right now. You don't need RAG.

Three Cases Where I'd Talk You Out of RAG

First, a tiny knowledge base. A few dozen pages, tens of thousands of words — stuffing the whole thing into the prompt is cheaper, more accurate, and has fewer failure points. In 2026, mainstream models start at 128K context windows; your "knowledge base" may be shorter than somebody's system prompt. Doing RAG for RAG's sake is manufacturing ops burden for yourself.

Second, data that changes by the minute and users who need the "current value". Live inventory, live prices, ticket status — this is "query the database", not "read the document". RAG's incremental update pipeline (split → embed → index → refresh) has inherent latency; using it for real-time queries is like driving nails with a screwdriver. The right move is to give the model API access or SQL queries, and save RAG for static knowledge.

Third, closed-domain Q&A. With only twenty or thirty fixed questions (pricing, refund policy, contact info), intent classification plus template replies will beat RAG on both accuracy and cost. Don't think "template" is lowbrow — users want the right answer, not cool tech.

One more thing: if what you're building is site search (users type keywords, you return a document list), that is not RAG Q&A — don't force this funnel onto it. When to search and when to answer questions is covered clearly in our in-site piece vibe-project-search-guide: tell the two apart before you start building.

Chunking Is the Number-One Lever: 80% of Retrieval Quality Dies Here

Conclusion first — also the single most important judgment in this piece: 80% of RAG quality is decided before retrieval, and most of retrieval quality is decided at chunking. Vector-store choice and rerankers, the fancy stuff, cannot rescue a shredded knowledge base. Chunks too big, and retrieval returns pure noise; chunks too small, and answers get cut off mid-thought, forcing the model to fill the gaps by inventing — "confidently fabricated answers" usually start right here.

A Sane Starting Config: 300–512 Token Recursive Chunking + 20% Overlap

Don't start with semantic chunking. It's expensive and its results are unstable — a one-person team should skip it for now. In 2026, the industry's safest starting config is:

• Chunk size of 300–512 tokens (roughly 2,000–3,500 Chinese characters): too small starves the context, too big drowns the signal in noise; this range is the sweet spot for most Q&A scenarios.
• Recursive splitting: cut down the paragraph → sentence → word hierarchy, breaking at natural boundaries instead of slicing hard at token 512.
• 20% overlap: adjacent chunks overlap by about 20% so key information doesn't land exactly on a split boundary. 0% loses boundary information, 50% wastes vector-store capacity and dilutes retrieval precision — 20% is the empirical balance point.

import re

def recursive_split(text, chunk_size=450, overlap=90):
"""递归切分：段落 -> 句子 -> 词，块大小约 450 token"""
paras = [p.strip() for p in text.split("\n\n") if p.strip()]
chunks, buf, buf_len = [], [], 0
for p in paras:
# 段落内再按句子切，避免超长段落撑爆 chunk
sentences = re.split(r"(?<=[。！？.!?])", p)
for s in sentences:
s_len = len(s) # 中文场景用字符数近似 token
if buf_len + s_len > chunk_size * 4 and buf:
chunks.append("".join(buf))
# 保留最后 20% 做 overlap
keep = "".join(buf)[-overlap * 4:]
buf, buf_len = [keep], len(keep)
buf.append(s)
buf_len += s_len
if buf:
chunks.append("".join(buf))
return chunks

Special Handling for Three Content Types: Headings, Tables, Code

A generic splitter handles body text fine, but these three content types need special treatment — otherwise they're reliable hallucination factories:

• Headings: prepend them as "anchor prefixes" on every chunk. When splitting, stick the parent h1/h2 heading onto the front of each chunk, like + body text. Otherwise retrieved chunks are headless, tailless text, and the model has no idea which topic it's answering.
• Tables: one chunk per table, never cut mid-table. Convert tables to Markdown text blocks and index them whole; if there are too many rows, group "header + a few rows" but carry the header into every group. Truncated tables are hallucination hotspots — the model will confidently stitch the two halves into a wrong conclusion.
• Code: split by function or class, never mid-body. The chunking unit for code docs should be function signature through function end, prefixed with the file path and function name. Feed the model a for loop cut in half, and the "usage example" it invents will make you want to laugh and cry.

Remember the order: get chunking right before you talk vector stores. Get chunking wrong, and every later step is premium renovation on garbage. A practical acceptance test: randomly pull 20 chunks and read them with human eyes — if more than 3 feel "headless and tailless", the chunking strategy goes back to the drawing board. No machine metric beats one human scan.

Vector Store Choice: A Solo Team Gets Three Tiers

Vector store selection is the easiest place in RAG to over-engineer. My judgment is blunt: a solo team doesn't need a fourth option. Read the three tiers in order and take whichever fits:

	pgvector	Self-hosted Qdrant	Managed Pinecone
Who it's for	Projects already on Postgres (Supabase users qualify automatically)	Large data volume, want control, willing to do ops	Don't want to touch ops, happy to pay for peace of mind
Cost	≈ 0 (reuse your existing database)	One VPS is enough to start	Usage-based billing, free tier at small scale
Hybrid retrieval	Native (tsvector full-text + vectors in the same DB)	Supported (sparse + dense vectors)	Supported (sparse vectors added in 2026)
Ops burden	None added	You own backups, upgrades, monitoring	Zero
One-line advice	Default to this — don't choose for choosing's sake	Move over at million-scale vectors or when you want advanced filtering	Consider when the team has zero ops capacity and real budget

The key point: if you're already on Postgres (Supabase, Neon, RDS all count), just add the pgvector extension — don't introduce a new component. The BM25-style full-text index that hybrid retrieval needs already exists in Postgres as tsvector; vectors and full-text live in the same database, and RRF fusion is a single SQL query (ready-made SQL in the next section). Every extra component is one more failure point and one more backup strategy — a solo team can't afford that.

Don't fuss over index params: start HNSW with the defaults (m=16, ef_construction=64); below million-scale vectors, tuning pays less than writing 10 more test questions. For embedding model selection, see our piece edge-local-inference-guide, which has measured comparisons of 2026 local embedding models — the one-line verdict: for Chinese Q&A, pick models optimized for Chinese; don't blindly take the top of an English leaderboard.

-- pgvector 表结构示例：文档 chunk 一张表搞定
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE doc_chunks (
  id          uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  doc_id      text NOT NULL,          -- 来源文档 id
  title_path  text NOT NULL,          -- 标题锚点，如"退款政策 > 未发货订单"
  content     text NOT NULL,          -- chunk 正文（含标题前缀）
  embedding   vector(1024),           -- 按你选的模型维度填
  meta        jsonb DEFAULT '{}'      -- 页码、URL、更新时间等
);
CREATE INDEX ON doc_chunks USING hnsw (embedding vector_cosine_ops);
-- 全文索引：混合检索的另一半
ALTER TABLE doc_chunks ADD COLUMN tsv tsvector
  GENERATED ALWAYS AS (to_tsvector('english', title_path || ' ' || content)) STORED;
CREATE INDEX ON doc_chunks USING gin (tsv);

Hybrid Retrieval: Don't Let Error Codes Slip Through Your Fingers

Pure vector retrieval has a hidden weakness: embeddings are insensitive to exact tokens. A user asks "how do I fix error code E_4031" or "the breaking change in v3.2.1", and vector retrieval may hand you a pile of "semantically related but utterly wrong" documents — because it understands semantic similarity, not "this exact string must appear". Product names, error codes, version numbers, person names: these exact-match terms are precisely what matters most in Q&A.

In 2026, the production default is already hybrid retrieval: BM25 (exact keyword match) + vectors (semantic match) recalling in parallel, fused with RRF (reciprocal rank fusion). Take top 50 from each lane, fuse by rank weighting — k=60 is the industry's standard empirical value. The principle isn't complicated: BM25 catches exact terms, vectors catch semantics, and RRF lets the two lanes correct each other.

-- Postgres 一条 SQL 做混合检索 + RRF 融合
WITH bm25 AS (
  SELECT id, ROW_NUMBER() OVER (
    ORDER BY ts_rank(tsv, plainto_tsquery('english', :query)) DESC
  ) AS rank
  FROM doc_chunks
  ORDER BY ts_rank(tsv, plainto_tsquery('english', :query)) DESC
  LIMIT 50
),
vec AS (
  SELECT id, ROW_NUMBER() OVER (
    ORDER BY embedding <=> :query_vec
  ) AS rank
  FROM doc_chunks
  ORDER BY embedding <=> :query_vec
  LIMIT 50
)
SELECT c.id, c.title_path, c.content,
       SUM(1.0 / (60 + COALESCE(b.rank, 999) )) +
       SUM(1.0 / (60 + COALESCE(v.rank, 999))) AS rrf_score
FROM doc_chunks c
LEFT JOIN bm25 b ON b.id = c.id
LEFT JOIN vec  v ON v.id = c.id
WHERE b.id IS NOT NULL OR v.id IS NOT NULL
GROUP BY c.id, c.title_path, c.content
ORDER BY rrf_score DESC
LIMIT 10;

One rule for tuning order: fix recall first, ranking second. Pull 50–100 candidates per lane to guarantee "the right answer is in the candidate set" — that's the foundation; ranking (rerank) is just ordering candidates, that's the interior decorating. Pretty decorating on a broken foundation buys you nothing — the root cause of many "RAG answers the wrong question" incidents is a top-k set too small for the right answer to ever enter the candidate set.

Reranking: Only Spend Money When You're Unsure

Cross-encoder rerankers genuinely improve ranking precision, but they're expensive: 100ms+ of extra latency and one more model call per question. For a solo team, my advice is conditional reranking — not every question gets reranked, only the ones where "the system isn't sure":

• Top-1 score too low: the hybrid retrieval's top RRF score sits below threshold, meaning the whole candidate set looks suspect — rerank to try to save it.
• Top-1 and top-2 too close: the top two scores are nearly tied, meaning the ranking is a coin flip — let the cross-encoder decide.
• The query has exact terms but the BM25 lane missed: e.g., the user asked about an error code but none appear in BM25's top 50 — then the semantic lane's results are probably all noise, reranking won't save it; go straight to refusal (next section).

def maybe_rerank(query, candidates, threshold=0.35, margin=0.08):
    """条件重排：拿不准才花钱"""
    if not candidates:
        return []
    top1, top2 = candidates[0], candidates[1] if len(candidates) > 1 else (None, 0)
    top1_score = top1["rrf_score"]
    gap = top1_score - (top2["rrf_score"] if top2 else 0)
    # 够确定：直接返回，省下 100ms+
    if top1_score >= threshold and gap >= margin:
        return candidates[:5]
    # 拿不准：上 cross-encoder 重排
    pairs = [(query, c["content"]) for c in candidates[:20]]
    scores = cross_encoder.predict(pairs)  # 精确但贵，只在此时调用
    ranked = sorted(zip(candidates[:20], scores),
                    key=lambda x: x[1], reverse=True)
    return [c for c, _ in ranked[:5]]

Don't copy my numbers — threshold and margin must be tuned on your own 30 golden Q&A pairs (see the RAGAS section below). Every knowledge base has a different score distribution; copying someone else's thresholds is running in their shoes. Also, if your store is small (a few hundred chunks) and recall is already solid, skip reranking entirely: complexity is cost, and the 100ms saved plus one fewer dependency buys you fewer 3 AM pages.

Anti-Hallucination on the Generation Side: Citation Labels + Daring to Refuse

Even perfect retrieval won't save you if the generation side is undefended — the model will still fabricate with a straight face. Anti-hallucination doesn't come from "begging the model to be honest"; it comes from two mechanisms: citation labels (every factual claim gets a source) and refusal (say "I don't know" when you don't).

Citation Labels: Every Factual Claim Carries a Number

Hard-code the rule in the system prompt: every factual statement the model outputs must carry a citation number [1] [2], mapping to retrieved chunks; sentences without retrieval support may not be written. The data structure looks like this:

// 检索结果喂给模型时的标准格式
{
  "query": "未发货订单如何退款？",
  "chunks": [
    {
      "ref": 1,
      "title_path": "退款政策 > 未发货订单",
      "content": "未发货订单可在支付后 24 小时内申请全额退款…",
      "source_url": "https://docs.example.com/refund#unshipped",
      "updated_at": "2026-09-20"
    },
    {
      "ref": 2,
      "title_path": "退款政策 > 已发货订单",
      "content": "已发货订单需在签收后 7 天内申请…",
      "source_url": "https://docs.example.com/refund#shipped",
      "updated_at": "2026-09-20"
    }
  ]
}

// system prompt 关键片段（直接抄）
"你是一个客服问答助手。规则：
1. 每个事实性陈述必须标注引用编号，如 [1]、[2]，编号来自上文检索结果；
2. 只能使用检索结果中的信息回答，禁止用训练知识补全；
3. 如果检索结果无法回答问题，直接说不知道，不要编造；
4. 引用编号必须真实存在，禁止虚构编号。"

Citation labels have a bonus benefit: users can click a citation to jump to the source text — trust goes through the roof. If you're building a help-center product, this mechanism and the knowledge-base construction in vibe-project-help-center-knowledge-base-guide are two sides of the same coin — that one covers "how to build the base", this one covers "how to do the Q&A".

Refusal: Turn "I Don't Know" Into a Scripted Template

Many vibe projects shy away from refusal, afraid the product looks "dumb". The opposite is true: the cost of one fabricated answer is trust that a hundred refusals can't buy back. At low confidence (retrieval scores broadly weak, or top-1 still below threshold after rerank), refuse outright, using a scripted template:

中文模板：
"这个问题我目前的知识库里没有覆盖到，我不想编一个答案糊弄你。
你可以：1）换个问法试试；2）直接联系人工客服（附链接）。
我已经把这个问题记下来了，会尽快补充相关文档。"

英文模板：
"I don't have this covered in my knowledge base yet, and I'd rather
not guess. You can: 1) try rephrasing; 2) reach out to human support
(link). I've logged your question so we can add the docs."

Refusal logs are the best source of product demand: once a week, look at the top 10 "questions users asked but we couldn't answer", and your doc-writing priorities are set. That's truer than any user research. For UX details around "the model inventing answers" (loading states, where the disclaimer goes, when to hand off to a human), our in-site hallucination UX guide covers it fully — no repetition here.

Run RAGAS Before Launch: The Four-Metric Minimal Eval Set

Launching RAG without evaluation is driving blindfolded. The good news: a solo team doesn't need an academic-grade eval stack — RAGAS's four core metrics plus 30 golden Q&A pairs, and you're running in half a day. The general methodology (how to build golden datasets, how to run regression pipelines) matches our in-site piece vibe-project-ai-evals-guide — that one covers the general framework for AI feature evals; here we cover only what's RAG-specific.

Four Metrics, Each Owning One Job

Metric	What it's asking	What a low score means
faithfulness	Does every sentence in the answer have support in the retrieved docs?	The model is inventing — go check chunking and citation labels
context precision	Among retrieved chunks, what's the useful-vs-noise ratio?	Retrieval is off — tune hybrid-retrieval weights and top-k
context recall	Were all docs needed to answer this question retrieved?	Something slipped through — check chunking boundaries and overlap
answer relevancy	Is the answer actually answering the user's question?	Wrong question answered — check query rewriting and the prompt

The four metrics map exactly onto the four stages of the funnel: whichever metric is low, fix that stage. That's the point of the "funnel" — evaluation isn't grading, it's locating which layer the fault is in.

30 Golden Q&A Pairs: All Three Question Types Required

The golden set doesn't need to be big — 30 pairs is enough — but all three question types must be covered:

• Answerable (15): answers clearly present in the knowledge base, covering heading/table/code content types; each pair labels the canonical answer and its source chunk.
• Refusal (10): no answer in the knowledge base; the expected behavior is to trigger the refusal script, not to invent. This is the anti-hallucination floor test.
• Edge cases (5): the answer is in the docs but phrased adversarially (e.g., asking for the third row of a table, or "besides A, what else"), testing retrieval and generation robustness.

# 黄金问答条目模板（golden_qa.jsonl，一行一条）
{
  "id": "qa-001",
  "type": "answerable",          // answerable | refusal | edge
  "question": "未发货订单如何申请退款？",
  "expected_answer": "支付后 24 小时内可申请全额退款…",
  "expected_refs": ["chunk-uuid-1"],
  "notes": "答案在'退款政策 > 未发货订单'chunk"
}

# RAGAS 评测最小集（伪代码，按官方文档装 ragas 即用）
from ragas import evaluate
from ragas.metrics import faithfulness, context_precision, context_recall, answer_relevancy

results = evaluate(
    dataset=your_golden_dataset,  # 30 条黄金问答跑出的 question/contexts/answer
    metrics=[faithfulness, context_precision, context_recall, answer_relevancy],
)
print(results)  # 哪个指标低，就去修漏斗的哪一层

One launch gate, and only one: if faithfulness drops below your own baseline, no launch. Note "your own baseline" — the first run's scores are the baseline; re-run after every chunking change, model swap, or prompt tweak, and roll back if scores drop. Don't copy the internet's "faithfulness must reach 0.9" line — scores aren't comparable across knowledge bases and question distributions, so copying someone else's threshold is meaningless.

Convergence: The Four-Stage Funnel Checklist

Finally, converge the whole piece into a pre-launch checklist for your project docs:

• Chunking: 300–512 token recursive splitting with 20% overlap; headings as prefix anchors; tables indexed whole; code split by function.
• Hybrid retrieval: BM25 + vectors recalling 50–100 each in parallel, RRF (k=60) fusion; guarantee recall first, then talk ranking.
• Rerank: conditional triggers, not every question; tune thresholds on your own golden Q&A, don't copy.
• Evaluation: RAGAS four metrics + 30 golden Q&A pairs (all three types); no launch when faithfulness is below baseline.
• Generation side: citation rules in the system prompt; refusal scripts at low confidence; review refusal logs weekly.

And the two most important judgments in this piece, repeated once more: 80% of RAG quality is decided before retrieval; if you can't answer "yes" to at least two of the four questions, don't use RAG. The biggest cost in tech selection isn't choosing wrong — it's paying ops costs for something you never needed.

Further reading: agent memory systems and document RAG get conflated constantly, but their retrieval targets, update cadence, and consistency requirements are completely different — the boundary is drawn in agent-memory-system-design. For the "is memory just RAG?" debate, Kevin Liao's discussion-sparking piece agent-memory-debate-kevin-liao-hn-october-2026 is worth reading with both sides in mind — it'll sharpen your thinking.