Building HKLawyer.ai: hybrid-search RAG over 20 years of case law

Legal questions are a good fit for retrieval-augmented generation. The answers already exist in judgments and ordinances; the hard part is finding the right paragraph and explaining it in plain language, in English or Chinese. This post covers how HKLawyer.ai does that, after 20,000+ questions from the public.

The data

The corpus is more than 20 years of Hong Kong Judiciary judgments. Judgments are long, structured documents: headnotes, facts, reasoning, orders. Naive fixed-size chunking cuts reasoning in half, so the first real decision was chunking by document structure rather than by token count.

Why hybrid search

Dense embeddings are good at meaning ("my landlord won't fix the roof" ≈ "duty to repair"), but bad at exact legal identifiers: ordinance names, section numbers, case citations. BM25 is the opposite. Legal questions need both, so every query runs through both retrievers and the rankings are fused:

const [dense, sparse] = await Promise.all([
  vectorSearch(query, { k: 20 }), // semantic similarity
  bm25Search(query, { k: 20 }), // exact terms, citations, section numbers
]);
 
// Reciprocal rank fusion: no score calibration needed
const context = rrf([dense, sparse], { k: 60 }).slice(0, 8);

Reciprocal rank fusion is simple, and it works because it only looks at ranks. Cosine similarities and BM25 scores live on completely different scales.

Measuring instead of guessing

"It looks right" doesn't scale. I built an LLM-as-a-judge workflow: a fixed set of 100 test queries, each answer graded against factuality and response-quality criteria, plus guardrails for questions the system shouldn't answer. The current pass rate is 80%, and the other 20% is a concrete to-do list instead of a vague feeling.

Designing for people who aren't lawyers

The retrieval is only half the product. The people asking are usually stressed and in a hurry, and they don't know the vocabulary. So the interface is built around a few rules:

• Plain language first, legal terms second. The answer explains the situation, then names the concept.
• Show the sources. Every answer points to the judgments it relied on, so people can check it and take it to a professional.
• Know the boundary. HKLawyer.ai gives legal information, not legal advice. When a question needs a lawyer, the answer says so.
• No account wall. Ask, read, leave.

Stack

• Next.js / React for the app and the SEO pages
• Python backend for retrieval and generation
• Qdrant / Chroma for dense vectors, alongside a BM25 index
• Docker deployment

Results so far

More than 20,000 questions answered for over 52,000 visitors since June 2025.

What I'd do differently

Build the benchmark first. I tuned retrieval for a while by eyeballing answers before writing the 100-query test set. Once the set existed, every change to chunking, fusion, or prompts had a number attached, and some of my "improvements" turned out to be neutral. Next time the evaluation set comes before the first prompt.

Treat bilingual search as a first-class problem. BM25 relies on tokenization, and tokenizers built for English don't segment Chinese text well. Dense retrieval hides some of that, but exact-term matching is exactly why hybrid search exists. I'd plan language-aware tokenization and evaluation for both languages from day one.

Check the judge, too. An LLM judge is fast and consistent, but it is confidently wrong sometimes. Regularly reading a sample of its verdicts myself keeps the 80% honest.

Cache the obvious. A lot of people ask variations of the same few questions: tenancy deposits, wrongful dismissal, small claims. Caching retrieval for frequent questions cuts latency and cost without hurting answer quality.

The biggest lesson is the least technical one: a RAG system is a product, not a pipeline. Retrieval quality, evaluation, and how the answer is presented all decide whether someone actually gets helped.