startupindex
07.2026A hand-rolled RAG system over 116 Indian startups, and the eval harness that decided which retrieval strategy shipped.
A RAG system over 116 Indian startups scraped from Wikipedia's unicorn list and Y Combinator's India directory, built without LangChain so that ranking, fusion and citation behaviour stay under direct control. Every claim carries an inline citation, and on a question the corpus cannot answer the model declines and names the source it checked.
architecture
Vector search and Postgres full-text search feed RRF fusion, then a BGE cross-encoder rerank, then streaming generation. Every claim carries an inline citation to the chunk it came from, and a question the corpus cannot answer gets a refusal that names the source that was checked rather than an invented answer. The cross-encoder costs 7,033 ms against 145 to 155 ms for every stage before it, so the interface streams each stage as it lands instead of waiting on the slowest one.
how i built it
- Skipped LangChain, so ranking, fusion and citation behaviour stayed under direct control.
- Automated the corpus build so the dataset refreshes in one call, scraping Wikipedia and Y Combinator, then deduplicating and embedding into Postgres.
- Tied every claim to an inline citation of the chunk it came from.
- Built a 41-question golden set spanning direct lookups, paraphrases, multi-hop questions and deliberately unanswerable ones, then measured all three retrieval modes against it.
- Streamed each stage as it landed, because the rerank costs roughly forty-five times what every stage before it costs.
results
- Plain vector search scored 0.839 hit@5, against 0.774 for hybrid with a cross-encoder and 0.613 for hybrid alone. The measurement chose what ships, and all three modes are published with the numbers behind them.
- An LLM judge scored faithfulness at 0.909 and abstention at 1.000 across the 10 unanswerable questions.
