Problem
I wanted answers to programming questions that I could check against the page they came from. The app answers from four open books: Pro Git, Eloquent JavaScript, Operating Systems: Three Easy Pieces and Jeff Erickson’s Algorithms.
Approach
Each question is searched two ways, by meaning with pgvector and by keyword with Postgres full-text search, and the results are merged with reciprocal rank fusion. A grader checks the top five passages. If they miss, the question is rewritten and retried up to three times before falling back to a web search. Every answer is cited and creates a clickable anchor which opens a drawer that lets you see the cited passage or the web page if it was cited from a webpage.
Stack
Next.js, LangGraph, Postgres with pgvector on Supabase, OpenAI, Tavily, Vercel.
Results
Recall@5 on my eval set was 93.8% for vector search alone and 90.6% for hybrid search, so adding keyword search didn’t help. I assume this was because of the way I set up keyword search and how poorly the pdfs were parsed given that the point of this project was more about building the RAG graph than building an ideal corpus.